Scope of this Report
This report identifies evidence that projects are late, over budget or deliver less than promised? It then considers various potential causes for these failures including culture, process, and estimation and how getting these things right can contribute to success.
What evidence is there that projects are late, over budget or deliver less than promised?
Most organizations develop business cases to initiate change1. These business cases require narrative explanation of the change and the associated financial return on investment.
Dan Galorath, noted software estimation expert, cites government data, “A recent US governmentreport showed 81% of budget or $57 billion in IT projects in danger of failing. Detailed reports on the hearings can be found here. Of 413 IT projects identified by OMB and federal agencies NEARLY 80% OFTHEM WERE IDENTIFIED AS HAVING BEEN POORLY PLANNED. The scorecard for IT projects shows muchprogress but much work left to do.”
The PMI Pulse Report 2014 pointed up some stark statistics. “Only 56 percent of strategic initiatives meet their original goals and business intent. This poor performance results in organizations losing $109 million for every $1 billion invested in projects and programs. High-performing organizations successfully complete 89 percent of their projects, while low performers complete only 36 percent successfully.”
Culture, Process or Estimation Issue?
Are process, culture or estimation responsible for the failures? Any or all of them can have a significant impact on a project’s performance. The tendency is always to blame the supplier for the failure – ‘Company X failed to deliver the ABCD project on-time for the XYZ government’, is not an uncommon headline. In truth it’s normally a combination of all three.
We need to look at potential sources of failure from several directions
- Culture: Is the organisation working as one towards a common, transparent, communicated goal?
- The Governance Process: What decisions need to be made, who makes them and are they tracked to completion?
- Backlog Prioritisation and Change Control: Was there a product backlog or its equivalent effectively managed and prioritised?
- Is estimation effective?: Are estimates based on facts or opinions?
- Is the development model effective?: Agile, Iterative or Waterfall methods are found, but are they effectively policed?
Weaknesses or failures in any of the above will put a project at risk. It is incumbent on both the supplier and Business teams to ensure that there are strong robust processes in place to de-risk the project.
The PMI Pulse report consists of responses to a voluntary questionnaire and is therefore self-selecting, but it is a valuable resource for discussion of what seems to be a constant refrain over the years.
The clear message of the report is that the most successful organisations in terms of project delivery have strong processes backed by effective measurement and project management offices.
Success comes on the back of success. Companies with effective traditional development methods can adapt quickly and effectively to agile methods. The key here is the word “effective” – Kotter suggests that without urgency, transformation cannot happen – and change is harder for some companies than others. The whole organisation adopts agile because of effective leadership, visible sponsors and a commitment to succeed. Such organisations are either consciously or culturally Lean. To them facts influence decisions; changes to process are tracked and monitored, and successful change remains while unsuccessful change is found early and discarded. During projects deviations from the norm are analysed and corrections are made. Projects seldom fail.
Contrast that with poor process organisations that change methods to follow the latest trends. For them a change in process is an excuse for chaos. Typically we see blame cultures, with poor communications and absent sponsors. Use of metrics is poor and often concentrates only on cash and time to market. If a project falls behind schedule the typical response is to throw people at it. Hordes of heroes are bound to help. Once again we have people repeating the same behaviour expecting a different result. That’s the definition of insanity often, incorrectly, attributed to Einstein. Whoever said it was right. In this instance insanity comes from not analysing reasons for failure but looking for quickv“obvious” answers, and doing that repeatedly.
Process – Governance
We look for an IT governance framework which has similar characteristics to the model proposed by Weill and Ross in 2004:
- Identify what decisions you need to make;
- Identify who makes those decisions – an individual;
- Identify how those decisions will be made e.g. what data is needed, who else should contribute opinions.
Effective organisations have clear governance based on effective leadership and visible sponsorship. They avoid committee decision making and clearly communicate decisions. Crucially they have effective monitoring and measurement activities so that deviations from course are made knowingly or are corrected quickly with little drama.
Process - Development methods
All development methods demand process. Some, such as waterfall, can be process intensive. Agile by contrast is process light, but it’s not process absent. Rather, we can say that Agile is less prescriptive.
Many effective organisations use waterfall or iterative development with defined methods backed by strong metrics and effective reporting. The best performers can adapt to agile when it fits the situation and they continue to be successful.
Agile works best when thought of as a lean process and that means that once you commit to build some user functionality you should do it only once. That means taking a disciplined approach to defining the minimum marketable features, refining the product backlog and delivering sufficient documentation to enable maintenance. It can become a game in ineffective organisations.
Two conversations, which we have had recently, underscore the need for discipline in the use of agile methods. In both cases the productivity related to delivery of functionality was defined as low. When probed, the reason is that both organisations were content to develop and re-develop the same functionality a number of times – to get it right in the end. The business and the development teams seemed to accept that delivering a business change using agile methods allowed for infinite changes of mind. This adoption of agile is not cost effective and gives rise to concern about how effectively agile methods are being used. Instead of the oft-quoted “paralysis by analysis” we see in waterfall, we have “endless enhancement,” and in either case a lot of time, energy, creativity and money is wasted.
Again the lesson is that effective development methods work to their maximum potential when the right amount of control and monitoring of progress is applied. Measurement and reporting is often seen to be an overhead and expensive. We have found that effective measurement and reporting consumes about 1% of project budget (1.5% to 2% on small projects and as low as 0.5% on major programmes). Companies that want to use facts to manage recognise this as money well spent as it enables effective management. Those that look for easy cost-cuts generally take out “overhead” first preferring to chart their course through the icebergs in the dark with a small rudder.
Estimating is difficult and the key thing to remember is that it is only an estimate. Often these become written in blood as the initial and final answer. Estimates should be a living thing throughout the project and should be revisited when either something significant changes in the project or when we know sufficiently more to refine the estimate.
In a waterfall development, estimates should at least be performed at Requirements, end of High level design, the start of Construction and reviewed at implementation. Just because Agile doesn’t have the traditional phases, early estimates are still important, and just as useful.
However, estimates can be revisited any time during the lifecycle such as when requirement shift or other variables come into play which will impact originally stated outcomes.
Failure to review and maintain project estimates means you can’t manage risk or use any contingency.
Early Estimates and the Challenges
Early in the project lifecycle, cost and schedule estimates are generated based on best information available. As is often the case, this information is lacking in detail and most likely is ambiguous. This presents several problems for the estimator. For example, ambiguous requirements make it difficult to determine a proper size.
However, by making and documenting stated assumptions, the estimate produced early in the lifecycle can be effectively managed and customer expectations can be properly set.
What Should Estimates Be?
Estimating is a risk assessment activity. The wise project manager can use a well-developed project estimate to properly set and manage end user expectations. Transparency is the watchword here. By sharing stated assumptions with the user and by helping them to understand the basis for the estimate, you are engaging them and making them share in the accountability for the estimate.
For example, if they are aware that the estimate is based on their requirements and the general feeling is that the requirements are somewhat incomplete then it can safely be assumed that another estimate will be required when more data is available and that the new estimate will probably be different from the original estimate.
“I want it delivered NOW!”
This dynamic shows itself, not so subtly, when management doesn’t really want an estimate at all; they want the software delivered when they want it delivered.
How many times have we seen a situation where the sales/marketing group, the business users or even our own senior management has requested a software solution that has a fixed delivery date already attached to it? And even though the user or senior manager may ask for an estimate they really aren’t interested in the response unless they are told what they want to hear or alternatively the supplier is told what the answer already is.
In this type of management environment, the IT organization doesn’t invest much time in their estimating practice because they don’t realize the power of good estimation as a vehicle to properly manage the project and/or their customer’s expectations. The net result is the best endeavours by the IT organisation in a blind attempt to deliver the requirement and usually a project starting with a high tariff of risk.
One perceived problem that is seen in expert estimates is that of memory. Estimates based on memory are subject to the cognitive bias of the estimator therefore involvement of others provides a balance that helps to cancel the potentially negative impacts of bias.
Single expert estimates tend to be either too high or too low depending on the estimator responsible and the culture, for example, the Scotty from Star Trek syndrome (bias) creeps into play with expert estimates. The seasoned estimator estimates high knowing it’ll be corrected anyway by the PM and the ultimate result may be a sensible figure.
The other typical scenario is it is easy for the expert to estimate how long a piece of work would take him to do but when he has to estimate for less experienced colleagues then it’s much more difficult to achieve so we tend to get under-estimates.
Don’t accept just a number for the estimate, three point estimates and the estimate assumptions are a key way to review and validate estimates.
Use of three-point estimating techniques also allows a more reasoned view of the estimate. In reality usually when we estimate we always think of a range –how long does it take you to get to work each day it might be 30 minutes on average but 20 minutes with quiet roads and 50 minutes in rush-hour. So combining all three estimates (Optimistic of 20, Average of 30 and Pessimistic of 50) gives you a much better view of the risk and what contingency you may need to use.
Expert estimates are a key estimate to obtain but there is great value in obtaining another estimate to reconcile this estimate against.
In the Agile space, the benefit of normalizing various experts estimates is often formalized through “planning poker” which constrains the estimate values that experts are allowed to choose and then requires the experts to justify and ultimately reconcile their estimates with each other. Given how effective it is in Scrum planning, the same process could and should apply more widely to expert estimates in general.
How Else Should We Address the Estimating Issue?
We can view popular estimation techniques through two separate perspectives: data and algorithms. First, from the perspective of experiential data / historical data and algorithmic / collaborative techniques, we see that many of the experiential based techniques leverage collaborative techniques to combat perceived weaknesses.
Historical data is used both in model based and expert estimates. Estimating without memory of the past is not possible. The bigger issue is whether models derived from historical data are clearly superior to expert estimates. If you are trying to remove the need for expert estimators the answer is unfortunately ... no.
Finally what is true is that expert estimates require a level of expertise that is sometimes not readily available, which then requires leveraging tools to be used to validate estimates. If estimates are important and the required level of expertise is not available then the choice is far starker. Estimates generated from models leveraging historical data in calibrated tools are the only logical choice.
Volatility and the Impact of Change?
Another key characteristic of a failing or delayed project is the degree of volatility and change. Studies show the exponential rising cost of change as you get in the later stages of a project particularly with a Waterfall methodology.
Agile is designed to accommodate change but change can occasionally be an excuse to use it as in, ”I don’t need to know what I want, I can keep changing my mind and we’ll be fine if we use Agile” can be a client view. This can lead to the same “priority” story being redeveloped multiple times until the client has worked out what they want and the Agile project fails because it runs out of time or money. Is the methodology at fault? Of course not but perhaps a requirements or design “spike” could have been implemented with the client to help them clarify their ideas.
Governance of change is the key, if you know what the business is going to look like at the implementation of the project then the project will control change and is much more likely to succeed.
Deviation from the Norm?
Often, changes to applications with regular release cycles tend to be of a similar size with the same team doing the work. The expert estimates roll into complexity matrices and sensible size metrics and all should be well as long as the estimators continually update their historical records and test the results against external databases. We still see the same mistakes repeated in the hope that a miracle will happen.
The challenge is when we deviate from the norm. Compressed timescale, significant increase in size will invalidate the current estimating methodology in the project. People will assume we can deliver at the same rate and the project is set to fail.
Lawrence H. Putnam published an empirical software estimation model in 1978. In the formula noted below, Size is the product size (whatever size estimate is used by your organization is appropriate). Putnam uses ESLOC (Effective Source Lines of Code) throughout his books.
- B is a scaling factor and is a function of the project size.
- Productivity is the Process Productivity, the ability of a particular software organization to produce software of a given size at a particular defect rate.
- Effort is the total effort applied to the project in person-years.
- Time is the total schedule of the project in years.
In practical use, when making an estimate for a software task the software equation is solved for effort:
Parametric estimating toolsets understand the likely impact and use this or their own bespoke calculation engines to deal with this and can make a major contribution in increasing the change of success in a project by setting realistic expectations.
Tracking and Monitoring
The final area to consider is effective tracking and control of the project. Continuous review of the projects velocity (Agile) or rate of delivery (Size measure per time period) will indicate the projects real status and chance of success, for example, if the development team report the project is 80% complete 3 weeks in a row then the project is likely in trouble.
Again the PMI report indicates that organisations that deliver successful projects, irrespective of the methods used, tend to have functioning and effective Programme Management Offices (PMO) and theses PMOs gather and analyse data effectively to support the successful delivery of business initiatives.
There is no need for projects to be delivered late, over budget or with less scope. These statements come from organisations that don’t understand that the light in the tunnel is actually a train coming full speed at you.
Successful delivery of a project requires a culture of effective business processes, effective estimation and sound development processes.
Strong business processes linking, effective business vision, realistic expectations and close communication between the vendor and supplier are key elements in successful delivery.
Development methods are only as good as the organisation that uses them. Whatever end of the process spectrum, it’s the effective use of the end to end processes that delivers the goods without drama.
The fundamental transformation of the idea to money comes with the estimate. Good estimates are living things that change with the circumstances.
Effective estimation requires an organization to commit resources to the development and execution of a well-defined software estimating practice, backed by a PMO that delivers effective data analysis.
- Dan Galorath on Estimating Blog, http://www.galorath.com/wp/government-it-project-woes-and-estimating-total-ownership-costs.php
- PMI Pulse Report 2014: http://www.pmi.org/Business-Solutions/Pulse.aspx
- “IT Governance: how top performers manage IT decision rights for superior results.” Peter Weill and Jeanne Ross 2004
- “DCG Works With Leading Customer Management Company to Implement Measurement and Governance Program for Data-based Decision Making” /insights/publications/measurement-program-for-data-based-decision-making/Is there a Business Case for Better Estimation? DCG Trusted Advisor Report, July 2013
- “What are the benefits if any of estimating my software projects through the use of a vendor developed estimating model?” DCG Trusted Advisor Report, July 2014
- Putnam Model http://en.wikipedia.org/wiki/Putnam_model