About 70% of IT projects fail. Best practices, skills and competencies have been improving for decades — so why do project still fail? Projects fail when, they solve the wrong problem, can’t find a good enough solution or over invest because of the psychological biases.
Table of Contents
The real reason why digital technology projects fail and how to fix it
Solve the right problem
Find a good solution to the problem
Summary
About 70% of IT / digital transformation projects fail. Approximately 45% are late and 25% get cancelled. The commonly held belief is that projects fail because they do not follow best practices and don’t have all the right skills on the team. However, the adoption of best practices, skills, and competencies have been improving for decades — so why do projects still fail?
The prevalent view is that that projects fail because they don’t follow best practices such as:
- Better control over changing scope and requirements
- Better project management practices
- Use of agile and short delivery cycles
- Managing strategy and stakeholders
- Better control over technology complexity
- Building better teams and improving communication
- More rigorous quality assurance
Projects that fail often have unclear objectives, shifting requirements, struggle to use technology correctly, suffer from poor team dynamics, or often missing critical skill sets and lack strong project management.
Such best practices are closely related with project success — but what if it’s only a correlation? What if improving best practices, does not improve project success rates, because they are not the true cause of project failures?
What if the true root causes are the following less explored reasons:
- Solving the wrong problem, or
- Not finding a good enough solution to the problem,
- Attempting to change systems without truly understanding how they work today and,
- Allowing psychological biases to influence investment decisions
Why best practices are not the root cause
Awareness of project methodologies have been evolving for decades. The number of skilled and certified people in the industry has been growing rapidly. The number of PMI certified project managers have been growing exponentially, so have the number of architects and business analysts.
Despite the growing capabilities and skills on project teams, the failure rate hasn’t budged much since the Standish Group and others began measuring this in 1994 — therefore it is logical to conclude that improving best practices and skills are correlated to project failure rate — but are not the root cause.
Improving best practices and number of certified professionals, does not improve project success rates.
We need to go beyond the common belief and dig deeper to find more likely root causes for why innovation projects (those attempting to implementing new ideas) and change projects (those replacing existing systems) are most likely to fail.
Possible Root Causes of Project Failures
Solving the wrong problem
Organizations and teams investing in innovation don’t spend enough time defining the most important problem they’re attempting to solve. Many have considerable difficulty even identifying which problems are crucial to their missions and strategies.Teams speed toward a solution, fearing that if they spend too much time defining the problem, their superiors will punish them for taking too long to leave the starting line.Many times, projects go down the wrong path. They end up implementing the wrong system and discover that they didn’t solve the problem they set out to fix in the first place. They only realize in hindsight the right problem to focus on and which path to follow.This results in shifting business requirements, missing deadlines, and re-work as teams go back to the drawing board. This eventually results in schedule and budget overruns.The rigor with which a problem is defined is therefore the most important factor in finding a suitable solution.
Most organizations are not proficient at articulating their problems clearly and concisely. Teams need to become a lot better at asking the right questions earlier and not starting initiatives too soon. They need to become proficient at using various problem-solving techniques such as the “Five Whys” to be sure they are tackling the right problems. They also need to spend more time earlier in the project analyzing if the economic cost benefit is viable before starting them.
Not finding a good enough solution
The next step after figuring out the problem, is to develop a solution. This requires building a minimum viable product (MVP) to learn what customers want as quickly as possible, then measure, and fine-tune the product until it reaches the tipping point.However, teams struggle to understand what constitutes a MVP and if what they have built is a good enough solution. Teams often get stuck in endless development cycles trying to meet the arbitrary standards of insiders, causing schedules to slip and budgets to balloon.The MVP is a version which allows teams to collect the maximum amount of validated learning with the least effort. The MVP is considered good enough when customers view it as a quality product and there is diminishing returns from further improvements that customers will never notice.
Developing the MVP is a fine art of architecting a solution that balances economics, customer-development, technical debt, and extensibility. Organizations and teams need to improve their use of techniques such as design thinking, prototyping, and user development research. They need to get better at validating if what they are building is useful to customers, much earlier.
Not understanding the current state
Projects aiming to transform existing systems by modernizing or replacing older systems fail because they often skip a major step — understanding the current state, “really really” well.Teams skip current state analysis or brush over it with the justification of starting the project faster. Current state analysis is viewed as a waste of time. Teams resist doing a current state analysis for many reasons including:
- Belief that what was done in the past was inefficient, no longer useful or relevant to the new reality
- Poor or lack of documentation on how the existing systems and processes work
- The original people who built the system or know most about it are no longer with the organization
- Data models and integrations are tricky to uncover
Skipping the current state step, often results in the new system falling short in many areas:
- The new system doesn’t support existing customers, products, services as well as the old system
- The new system doesn’t create the correct data
- Users resist adopting the new system as it does less than the old system did
- The new system doesn’t work correctly with the existing internal systems or external partners
All such issues result in cost and schedule overruns as teams have to re-analyze and re-implement the gaps. Organizations need to give teams the time and budget to do it right and teams need to get better at:
- Correctly understand how the business operates today
- Fully appreciate the conveniences built over time into the old system
- Reverse engineer existing data models
- Analyze upstream and downstream integrations
For most transformation projects, unless the business environment or business model has changed, some or all the existing capabilities will be needed in the future. The capabilities may take new form, but their essence may remain and must be well understood.
Challenges estimating enterprise IT project schedules and cost
The root causes explored so far offer reasons for why projects are late, but they don’t explain why 70% of projects are consistently over budget by approximately 40 to 50%. The average has been hovering close to 43%, since the Standish group started tracking this in 1994.
But why is the range “40% to 50%”?
When teams discover that more work is needed, the tendency is to request more funding to make it better. Funders become locked-in and entrapped in a course of action and resort to ‘throwing money’ to make the venture succeed. This is a psychological bias, known as the escalation of commitment to a course of action. Results from related psychological studies [968, Knox and Inkster[35], Staw and Fox[38] ] show that people will continue to invest approximately 40% more than they originally planned when they have a sense of personal responsibility, to avoid sunk costs, or desire not to appear wasteful.
But how far will funders go before they turn off the funding tap?
Over-funding projects will continue until the second psychological bias of loss-aversion kicks in and “the losses start to loom greater than the possible gains”. The idea of loss aversion is ingrained in the human mind and most people will feel the pain of loss twice as much as the pleasure of gains — explaining why the pain will become psychologically painful when overruns start to approach 50%.
The Ying-Yang effect between the escalation of commitment and loss aversion causes projects to overrun on average 43%. Such schedule slippage and budget overruns are often attributed to poor project management discipline, rather than to these fundamental psychological biases.
If the root cause of project cost overruns is psychological and has remained steady for decades, we can begin to develop solutions based on the reliability of this metric. The solution to project budgeting might in fact be to encourage teams to find solutions to problems by giving them constraints of 50% less budget or time, than what is available.This solution would work as follows:
Example 1: with no constraintsA project has a budget of $100,000. The project manager assigns the entire $100,000 to the team building the system. The team proceeds with the project and towards the end of the project, they discover they solved the wrong problem, or did not develop a good enough solution to the problem. They escalate this to the management team and present a proposal of how to improve the system and the cost to do so. The organization decides the improvements have a high likelihood of succeeding and provides an additional $50,000 in funding. The team completes the project and the total cost of the project is $150,000 — an overrun of 150%.
Example 2: with constraintsA project has a budget of $100,000. However, in this case the project manager assigns 50% less budget (approximately $66,000) to the team building the system. The team proceeds with the project and towards the end of the project, they discover they solved the wrong problem, or did not develop a good enough solution to the problem. They escalate this to the management team and present a proposal of how to improve the system and the cost to do so. The organization decides the improvements have a high likelihood of succeeding and provides an additional $33,000 in funding. The team completes the project and the total cost of the project is $100,000 — the project comes in on budget.
Conclusion
Despite the increasing maturity of software development methods, and the growing number of certified professionals, software project budgets and schedules are likely to continue to be very challenging to predict accurately.The root cause of projects over-running are human issues:
- Solving the wrong problem, or
- Not finding a good enough solution to the problem,
- Attempting to change systems without truly understanding how they work today and,
- Over investing because of the phycological biases of escalation of commitment and loss aversion
Projects can be delivered on time and on budget by directly countering the root causes:
- Finding the right problem to solve and finding a good enough solution — using design thinking, prototyping and customer development techniques
- Getting a deep understanding of the how systems work today — using current state analysis techniques
- Controlling the urge to over invest. by doing cost–benefit analysis and encouraging teams to find a solution to the problem to a constraint of 50% less budget / time than what is available.
