A CRITIQUE OF INITIAL BUDGET ESTIMATING PRACTICE

Transcription

1 A CRITIQUE OF INITIAL BUDGET ESTIMATING PRACTICE Sidney Newton The University of New South Wales, Australia Budget estimating practice has not changed fundamentally since cost planning was introduced some five decades ago. It continues to provide reasonable accuracy and confidence in the budgets for the majority of projects. As new projects seek to do things differently, on a different scale and unlike almost anything that has gone before, the practice of budget estimating based on traditional cost planning principles must be called into question. This paper presents a critical review of current initial budget estimating models and highlights the extent of the uncertainty and the lack of improvement in estimating accuracy over time. The argument that technical explanations (estimating error) or psychological explanations (optimism bias) are the basis for inaccurate cost estimates is rejected. Instead, a more objective and simplified statistical model is outlined reference class forecasting. KEYWORDS: initial budget estimates, estimating accuracy, reference class forecasting. INTRODUCTION Cost overruns, time delays and contract terminations are likely to be a growing challenge for the construction industry in the current economic climate. Fortunately, we have a generally well-established process of cost planning specifically developed to contain and control design, procurement and construction decisions effectively in project cost terms (Ferry et al, 1999). Around that broad process of cost planning has grown a range of cost estimating techniques, modifications and extensions to the basic cost planning process, and a wealth of the standardised cost data necessary to satisfy the basic principles of cost planning. The basic principles of cost planning were established at the outset, requiring: a frame of reference; a method of checking; and a means of remedial action (RICS, 1969). These principles have generally persevered, and cost planning is now recognised as an important management tool for any significant construction project. However, projects develop through various stages from inception to completion. Smith and Jaggar (2007, p.81), for example, use the RIBA Plan of Work to illustrate how the application of cost planning changes depending on the particular stage the project has reached, specifically: Pre-stage A: Establishing the Need establish the budget. Stage A: Options Appraisal cost of preferred solution. Stage B: Strategic Briefing target cost. Stage C: Outline Proposals prepare initial cost plan. Stage D: Detailed Proposals firm cost plan. Stage E: Final Proposals cost checks, design against cost plan. Stage F: Production Information final cost checks of design against cost plan. The stages listed here represent but one way of breaking down the pre-construction process, and are not necessarily discrete stages in any particular sense. However, they usefully 271

2 illustrate how the focus and outcome of cost planning does change over the course of a project. In particular, that the nature of the cost planning process in the early stages (when the specifics of a project are not clearly known) is distinctly different to that of the later stages (when more accuracy and higher fidelity is both required and possible). Of particular interest to this paper is the very earliest stage. In Plan of Work terms, the Prestage A: Establishing a Need. In any terms, however, the focus at that stage of the cost planning process is on the establishment of an initial budget estimate figure. It is at this earliest stage of the project that the specificity of what is to be constructed is at its broadest and most poorly defined, and the accuracy of the cost estimating technique likely to be at its most variable. Ashworth (2008, p.251) presents the usual levels of cost estimating accuracy achieved in practice, across all construction types, as follows: Table 1: Estimate Classification and Accuracy (from Ashworth, 2008, p.251) Estimate Purpose Accuracy Order of magnitude Feasibility studies 25-40% Factor estimate Early stage assessment 15-25% Office estimate Preliminary budget 10-15% Definitive estimate Final budget 5-10% Final estimate Prior to tender 5% In other words, at the earliest stage there is the maximum potential discrepancy between the budget figure estimate and the final project cost, and this discrepancy can be between 25 and 40%. What can be confusing about such accuracy predictions is the lack of consistency in what actually is being estimated: is it the lowest tender price, the total construction costs, or the overall project costs. Clearly, the accuracy variations will change depending on how far into the overall project the cost estimate is forecasting, but the earliest stage/maximum potential discrepancy relationship will always tend to hold. There have been several studies seeking to determine which factors most contribute to such a substantial potential variation in cost, or the accuracy with which such cost variation can be estimated (see for example, Aibinu and Pasco, 2008; and Serpell, 2005). The focus of this paper, however, is not so much on the factors that contribute to problems of accuracy, but rather, critically to review the alternative cost estimating approaches typically employed at the earliest stages of a project: the, so-called, Initial Budget Estimate models in current use. The objective of the review is to determine which cost estimating approach might best be used in the earliest stages of a project, when the intention is to estimate the overall project costs, or Outturn Cost. Determining the utility of competing budget estimating methods is now critically important, not least because project uncertainties at the earliest stages are unlikely to be mitigated any time soon. Indeed project uncertainties and are most likely actually to increase: economic conditions have become extremely volatile; the complexity of construction is increasing progressively as projects are required to become more innovative in order to satisfy a new order of constraints and expectations from environmental and social perspectives; financing 272

3 and risk management margins are being tightened, and the relative significance of traditional cost drivers has become almost chaotic and certainly unstable among various other factors also likely to increase uncertainty. INITIAL BUDGET ESTIMATE MODELS IN CURRENT USE It is, generally speaking, the purpose of any budget estimating model to provide as realistic an indication as possible of the final project cost (Outturn Cost), for a given project proposal. Every current form of model begins by generating an initial, best estimate of the final project cost (Base Estimate) for the project as it is conceived at that stage of the project development. The different models then employ different methods to estimate an amount to be added to the Base Estimate to allow for the inevitable uncertainty that is inherent in any estimating process. This amount is then the Uplift which is applied to the Base Estimate. Thus, ideally: Base Estimate + Uplift = Outturn Cost The three (3) generic model types most typically being used for budget estimating purposes are: Model 1: Deterministic Contingency Allowances Model 2: Quantitative Risk Analysis Model 3: Optimism Bias The Base Estimate is determined using one of a variety of preliminary estimating techniques, such as: functional unit pricing (where the cost is calculated on a per bed, per student, or other operational unit basis); area unit costing (where the cost is calculated on a per square metre of gross floor area or other measure of space provision); and/or elemental cost estimating (where consistent building elements such as foundations, frame, roof, etc., are costed individually). Model 1: Deterministic contingency allowances This is the traditional approach and variations of this model are used extensively. The level of experience and estimating competence are always important factors in the accuracy of the Base Estimate. They are particularly critical factors in determining the Uplift to apply in adjusting the Base Estimate for uncertainty using a deterministic contingency allowance. In this model the Uplift is determined through expert judgement and applied as a deterministic contingency added to the deterministic Base Estimate. Three (3) major types of contingency allowance are normally considered: estimating inaccuracy an allowance is added for the potential inaccuracy of the Base Estimate due to factors such as the lack of precedents and relevant cost data, the experience of the estimator, etc. Indications from a recent survey (Aibinu and Pasco, 2008) suggest that a contingency allowance of about 20% of the Base Estimate should be applied to take account of potential estimating inaccuracy in general. project uncertainty an allowance is added for the inevitable adjustments that occur as a project develops and becomes more resolved. At an early stage the Base Estimate may be generated on little more than a functional specification and a range of assumptions. Those assumptions are tested as specific scope, design and construction decisions are made. In line with the figures generated by Ashworth (2008, p.251), in 273

4 addition to the allowance for estimating inaccuracy, project uncertainty might require a further contingency allowance of up to 20% (maximum range of 40%, less 20% for estimating inaccuracy) of the Base Estimate. project complexity the contingency allowances for estimating inaccuracy and project uncertainty presented above are factors for all projects. The inherent complexity of a project has an aggravating effect on these normal factors. For particularly large, complex, innovative and high-profile projects a further contingency allowance may be appropriate. Given the other Uplift components are taken as averages, and complex projects tend to be at the top-end of potential cost overruns, an additional contingency for known project complexity of up to 35% might be reasonable. In very broad terms, that figure is determined from other research reported by The Allen Consulting Group (2007), The Tax Payers Alliance (2007), and Flyvbjerg et al (2002). It is apparent from the above that at the preliminary stages of a project, an overall contingency allowance of perhaps 75% (20% + 20% + 35%) of the Base Estimate might reasonably be applied. As the scope, design and construction details of a project develop, the level of confidence in the estimating accuracy and project certainty will improve, and the impact of project complexity will be mitigated. This method has the benefit of being relatively simple to apply and having a broad base of application. However, it does depend directly on the level of experience in the cost estimating technique and competency in risk analysis and management procedures. In any event it does remain a highly subjective approach, susceptible to inconsistent or biased judgement on the level of contingency to apply. The bias can be reduced by having multiple experts or an experienced team providing varied and independent opinion. It is problematic where a high level of confidence is required of the Initial Budget Estimate, as there is no indication on the degree of variation that might occur around the averages used. Thus, ultimately it is a deterministic (single figure) approach and, given the vagaries that attend the determination of an appropriate contingency amount, significantly higher levels of Uplift may be necessary where higher levels of certainty are required. Model 2: Quantitative risk analysis This model is used extensively in many government agencies and other large clients with consistent and ongoing construction needs. Its use has grown significantly in recent years, especially on particularly large-scale and/or complex projects. Sometimes it is possible to anticipate the circumstances that give rise to variability in outcome deterministically, but most often only a range of possible outcomes can be anticipated. In those circumstances the Base Estimate needs to incorporate an assessment of the risk and the confidence that attends such a forecast. The most significant benefit of making an explicit assessment of risk and confidence is that effective sensitivity analysis and planning control are then possible. This can further mitigate the inevitable uncertainty that attends a budget forecast. The most common form of quantitative risk analysis employs a three-point estimating practice based on Monte Carlo simulation (MoD, 2007). In that case, rather than a single cost estimate for each element of the overall Base Estimate, each element estimate comprises a range of possible costs defined by a Minimum and a Maximum, with the Most Likely element estimate located somewhere between those two extremes. The Minimum is an optimistic estimate of what might be the outcome if everything goes as well as could be 274

5 hoped. The Maximum is a pessimistic estimate that assumes what can go wrong, will go wrong (within reason). The Most-Likely estimate is the single figure estimate otherwise included in a deterministic estimating approach. The output of this method provides a cumulative probability (s-curve) distribution of the potential Outturn Costs. In most cases the distribution can be analysed further to determine the key factors driving the distribution (and variability). The actual Initial Budget Estimate is then selected from the probability distribution depending on the level of confidence expected/required at that stage of the project and the risk aversion of the budgeting authority. Typically: 10% Confidence represents a lean estimate, achieved only if most of the likely risks fail to materialise 50% Confidence represents the best estimate given that some risks will materialise and some will not, and 90% Confidence represents a value that could be reached if several unlikely risks all materialise together. This method has the benefit of providing a robust analysis and planning tool for risk management. It provides an explicit measure of confidence in the Initial Budget Estimate and indicates the general level of uncertainty applicable to that figure. More so even than for Model 1, it is critical that the quantitative risk analysis method be developed by a team of individuals and stakeholders, each bringing a slightly different perspective and expertise to the three-point estimating process. In addition, the technique does demand a specialist competence, training and experience to be applied effectively as with all statistical techniques, poor quality input data leads to poor quality output. It is also important that the method be employed as part of a broader framework of project risk management. There is the danger of biases and correlations going unnoticed in this technique, as it is a relatively blackbox process. However, these concerns are best allayed through correct application of the technique and a process of effective sensitivity testing. Model 3: Optimism bias Optimism Bias is a concept used to explain the tendency of people to be over-optimistic about the outcome of planned actions (Flyvbjerg et al, 2002). This includes over-estimating the likelihood of positive events and under-estimating the likelihood of negative events. Optimism bias arises in relation to estimates of costs and benefits and duration of tasks. It must be accounted for explicitly in appraisals, if they are to be realistic. Optimism bias typically results in cost overruns, benefit shortfalls and project delays. The UK government explicitly acknowledges that optimism bias is a problem in planning and budgeting (HM Treasury, 2003), but different departments have differing interpretations on the level of optimism bias to apply. For example, DEFRA (Dept of Environment Food and Regional Affairs) encourage the use a 50% optimism bias; Lancaster City Council use 60% initially; and Northern Ireland Building Procurement recommend an allowance of 35-40%. The UK Treasury itself has set optimum bias uplifts for projects as a whole and broken down for a variety of project characteristics. For smaller, less complex projects the optimism bias can be used directly as a general risk adjustment (as a direct alternative to the deterministic contingency allowances of Model 1). In all other situations the calculated optimism bias is 275

6 used as a top-down assessment of risk, and compared directly with a bottom-up calculation of risk from one of the quantitative risk analysis techniques of Model 2. Data collection from past experiences is an essential part of the optimism bias calculation process. HM Treasury (2003) suggests budgetary adjustments be based on experiential data from past projects and adjusted as required for the unique characteristics of the target project. For example, Figure 1 illustrates how an optimism bias adjustment to the Base Estimate is constructed and compared with an s-curve derived using some form of quantitative risk analysis model. Ideally, the Initial Budget Estimate so determined should be in the immediate vicinity of the 50 percentile confidence figure indicated by the s-curve. A traffic light marking system has been developed to aid this comparison. If the comparison result does not lie within the "Green" zone of the traffic light, then all calculations and assumptions should be reviewed. TRAFFIC LIGHT COMPARISON INITIAL BUDGET ESTIMATE Optimism Bias Uplift Base Estimate 10% 50% 90% CONFIDENCE LEVEL Figure 1: An Optimism Bias Uplifted Base Estimate Comparison (based on MoD, 2008) The advantages of this method are that it is relatively simple to understand and provides a consistent basis for comparison and benchmarking. However, the quality of the result directly reflects the accuracy of the tables of figures established for optimism bias. As indicated above, recommended values for optimism bias do vary and they will, in any case, require a large and robust data set to get even approximately right. As with all models, the final application is improved to the extent that empiricism and expertise are applied to the mitigating factors. The danger is always that such apparently simple models become more tempting for inexperienced users to apply. Guidelines are therefore particularly important with the optimism bias approach. Discussion of the current models The basic tenet of an effective cost planning process is to give primacy to the specific project under consideration. The more that is known about the current project (specific features and circumstances), then the more accurate the cost estimate is likely to be and the more effective the cost planning process becomes as a consequence. What this has tended to do is to 276

7 privilege traditional methods, where there is a familiarity with the process and its various tolerances and where extensive data sets already exist. But what is clear from the model descriptions provided above, is that the degree of variation around the potential contingency allowance, cost uncertainty and optimism bias figures used is substantial. If the Uplift amount might vary from 25%-75% for innovative projects under Model 1, or 35%-60% under Model 3, then the growing economic uncertainty is just further cause for concern. According to the Tax Payers Alliance (2007) a UK lobby group the total net cost of overruns in 305 public sector capital projects for the preceding 2 years was approximately US$35 billion. Despite the advent of improved estimating techniques and models, however, many studies conclude that estimating accuracy has improved only slightly, if at all (Aibinu and Pasco, 2008; Cheung et al, 2008; Flyvbjerg, 2007a). Flyvbjerg (2007a) goes on to reject the arguments that claim technical explanations (estimating error) or psychological explanations (optimism bias) are the basis for inaccurate cost estimates. Instead, Flyvbjerg shows that strategic misrepresentation on political and economic grounds (such as knowingly underestimating costs in order to gain national funding) is the stronger driver of poor estimating performance. What this suggests is that the fundamental premise of budget estimating as currently applied, where Uplift is determined through informed and specific insight into the particular project and its immediate context, might itself represent the real barrier to improved cost estimating performance. Could there be a tyranny of insight in operation here? It is unlikely (if not implausible) that the real culprit here is the application of specific project knowledge. More reasonably, if the genuine driver of poor budget estimate performance is a political-economic one, the real culprit may be the attachment to the project and its funding success that typically accompanies a more intimate understanding of the project. What may be required is a budget estimating model that displaces the need for insight and specific project knowledge with objectivity and distance. A FUNDAMENTALLY DIFFERENT INITIAL BUDGET ESTIMATE MODEL FOR MAJOR PROJECTS The clear distinction of this method is the way in which it provides an outside view on a specific project (how it sits within a context of a similar set of previous projects), where other methods look at a project from the inside (taking account of the specifics of the project, be that from a top-down or a bottom-up perspective). The advantage of this outside view is that all of the subjective elements associated with other methods of calculating an appropriate Uplift are dispelled. Being a statistical approach, a significant number of similar previous projects have to have been analysed in terms of their cost overruns, which may be particularly problematic for projects in smaller construction markets. It is also the case that as the project progresses, typically budget uncertainty is reduced. In order to arrive at a valid Total Budget Estimate at a later stage than the preliminary stage, Uplifts should normally be adjusted progressively downwards. According to Flyvbjerg (2007b), The Dutch econometrician Henri Theil (1961) liked to study the accuracy of forecasts. Once he did a study of accuracy in Dutch weather forecasts. Much to his surprise he found that if the weather forecasting service, instead of doing its forecasts, had simply played an automated recording stating every day that "the weather 277

8 tomorrow will be like the weather today," then the accuracy of predicted weather would have been higher than the accuracy actually achieved with the weather forecasts. Extending this anecdote, Flyvbjerg et al have proposed a fundamentally different initial budget estimating model from those currently used. Effectively, the model takes the position that the Uplift required to be applied to a new project will be like the Uplift that characterises recent projects of a related type. The model is called Reference Class Forecasting. Model 4: Reference class forecasting This model is a recent development in Europe, and was recently endorsed by the American Planning Association. Reference class forecasting takes a statistical view of any given project, based on a comparison of the Initial Budget Estimate and the final Outturn Costs derived from a set of related previous projects. It does not try to predict specific uncertainties, but rather simply places a given project in the statistical distribution generated from the reference set. Any approach that calls on human judgement and intervention in the estimating process is going to be biased in some direction and to some extent. Reference class forecasting, by making the assessment objective and statistical in nature, is a method for removing the bias from an Initial Budget Estimate or at least the Uplift component of same. Reference class forecasting requires the following three steps in order to determine the most appropriate Uplift to apply to a given Base Estimate: a relevant set of reference projects are identified from past data sources. The set must be broad enough to be statistically meaningful but narrow enough to be truly comparable with the particular project under consideration. a probability distribution for the selected reference set is generated. This requires access to credible, empirical data for a sufficient number of projects within the reference set to make statistically meaningful conclusions. comparing the specific project with the reference set distribution, in order to establish the most likely Uplift to apply to the particular project. Undoubtedly simple statistics, based on historical precedent, will fail to predict the extreme outcomes that lie outside the original set of precedents. However, for the vast majority of other cases the statistical approach will produce more accurate results because the model avoids having to allow separately for psychological and systemic biases such as optimism bias. Figure 2 illustrates how a reference class forecast is used to uplift a Base Estimate. The reference class forecast is presented in the form of an s-curve that represents the actual cost overruns as a percentage of the original Base Estimate, determined from a reference set of past projects. The Initial Budget Estimate figure is then determined by selecting the required level of confidence (percentage of projects within a given cost overrun) and reading off the required Uplift from the actual cost overrun axis. This Uplift is then applied to the current Base Estimate. For example, in 2004 the first instance of reference class forecasting was used by Ove Arup to demonstrate that an original budget estimate of 320 million for a proposed Edinburgh Tram Line project was likely to be conservative. Their forecast at the 50 th percentile was 357 million (Flyvbjerg, 2006). 278

9 ACTUAL COST OVERRUN AS A % OF BASE ESTIMATE (Uplift) 60% 45% 30% 45% Uplift required for 50% Confidence 0% -10% Base Estimate 0% 50% 100% PERCENTAGE OF REFERENCE SET PROJECTS WITHIN (LESS THAN) THAT COST OVERRUN Figure 2: A Reference Class Forecast CONCLUSIONS A critical review of current initial budget estimating models has highlighted the extent of the uncertainty and the lack of improvement in estimating accuracy over time. There seems no end to the growing number and substantial financial impact of project cost overruns. At a time when uncertainty is increasing, other models must be considered. The argument that technical explanations (estimating error) or psychological explanations (optimism bias) are the basis for inaccurate cost estimates has been rejected. Instead, a more objective and simplified statistical model has been outlined reference class forecasting. The clear distinction of reference class forecasting is the way in which it provides an outside view on a specific project, where other methods look at a project from the inside. The advantage of this outside view is that all of the subjective elements associated with other methods of calculating an appropriate Uplift are dispelled. The limitations of this approach, being a statistical approach, are that a significant number of similar previous projects have to have been analysed in terms of their cost overruns. This may be particularly problematic for smaller construction market contexts. It is also the case that the approach only works when the bias that causes the inaccuracy is not deliberate. In that circumstance the potential for reference class forecasting is low. 279

10 REFERENCES Aibinu, A. A. and Pasco, T. (2008) The accuracy of pre-tender building cost estimates in Australia, Construction Management and Economics, 26(12), Ashworth, A. (2008) Pre-Contract Studies: Development Economics, Tendering and Estimating (3 rd Edition). Oxford: Blackwell Publishing. Cheung, F. K. T., Wong, M. W. L. and Skitmore, M. (2008) A study of clients' and estimators' tolerance towards estimating errors, Construction Management and Economics, 26(4), Ferry, D. J., Brandon, P. S., and Ferry, J. D. (1999) Cost Planning of Buildings (7 th Edition). Oxford: Blackwell Science. Flyvbjerg, B. (2006) From nobel prize to project management: getting risks right, Project Management Journal, 37(3), Flyvbjerg, B. (2007a) Policy and planning for large-infrastructure projects: problems, causes, cures, Environment and Planning B: Planning and Design, 34, Flyvbjerg, B. (2007b) Truth and Lies About Megaprojects: Inaugural Speech, available at: Flyvbjerg, B., Holm, M. S. and Buhl, S. (2002) Underestimating costs in public works projects: error or lie?, Journal of the American Planning Association, 68(3), HM Treasury (2003) The Green Book: Appraisal and Evaluation in Central Government, London: TSO. MoD (2007) Three-Point Estimates Overview, available at: MoD (2008) Policy, Information and Guidance on the Risk Management Aspects of UK MOD Defence Acquisition: Version November 2008, available at: RICS (1969) Building Cost Information Service (BCIS) Standard Form of Cost Analysis: Principles, Instructions and Definitions. London: Royal Institution of Chartered Surveyors. Serpell, A. F. (2005) Improving conceptual cost estimating performance. AACE International Transactions, EST.13, 1-6. Smith, J. And Jaggar, D. (2007) Building Cost Planning for the Design Team. Oxford: Elsevier. The Allen Consulting Group (2007) Performance of PPPs and Traditional Procurement in Australia: Final Report. Available at: The Tax Payers Alliance (2007) Beyond the Dome: Government Projects 23 Billion Over Budget, available at: 280