Meta-metrics for the Accuracy of Software Project Estimation

Transcription

1 Meta-metrics for the Accuracy of Software Project Estimation T.L. Woodings Department of Information Technology, Murdoch University and Comast Consulting Pty Ltd PO Box 88, Nedlands, Western Australia 6009 Phone: (618) Abstract Software project estimation for such items as Size, Effort, Cost, Delivery time, Reliability and Risk, is a fundamental skill for software engineers. In order to improve estimation, there is a need for measurement of the measurements, that is, a meta-metric for the process. This paper compares three existing metrics and provides an example of conversions between them. It then extends one (DeMarco's Estimating Quality Factor) to recognise and reward rapid convergence to an accurate figure during a project. A second metric is defined to provide a lower bound on error for multiple initial estimates. The paper concludes with a worked example as an illustration of the metric's properties. 1. Background One of the major problems of Software Engineering is the lack of confidence in estimates of project factors such as Size, Effort, Cost, Delivery time, Reliability and Risk, even though these are of fundamental interest to the client. Poor estimates lead to poor plans and inadequate planning is a basic cause of failure with software projects. Despite substantial developments in recent years in the field of Software Metrics, industry's ability to predict basic project parameters is low and the proportion of systems delivered significantly overdue remains unacceptably high. A previous paper (Woodings, 1995) considered a taxonomy of Software Metrics with particular reference to giving greater visibility to measures of process improvement. This paper considers the issue of meta-metrics for the accuracy of measurement of project parameters in general (and development effort in particular) and the need for specialised metrics which may be monitored to provide evidence of organisational improvement. The need for meta-metrics was emphasised in some recent research (Lederer, 1998), which indicated little improvement in estimation practices due to new models and techniques. It asserted: "Only one managerial practice - the use of the estimate in performance evaluations of software managers and professionals - presages greater accuracy. By implication, the research suggests somewhat ironically that the most effective approach to improve estimating accuracy may be to make estimators, developers and managers more accountable for the estimate even though it may be impossible to direct them explicitly on how to produce a more accurate one." However, the more managers are held to their promises, the more they will be tempted to adjust other project factors in order to look good. Predictions become self-fulfilling prophesies. Thus, there is a need to have in place a framework of meta-metrics to monitor, guide and provide feedback to organisations on their project management.

2 T L Woodings 2. Existing Metrics for Project Estimation Accuracy The accuracy of an estimate is given by Relative Error R R = E - A / A where E is the estimate and A is the actual result at the conclusion of the project. There are three metrics in general use for the measurement of accuracy of project parameter estimation (Fenton, 1997). All three may be used for any parameter and are independent of the units of measurement. (i) In a study of three major estimation techniques, Kemerer (1987) uses mean percentage magnitude of relative error MPMRE over n projects: MPMRE = 100Σ R / n As an example, Kemerer's data gives the MPMRE for estimating effort over 15 projects using Albrecht's Function Points as , with for the standard deviation. (ii) A non-parametric alternative is a binary (success or failure) measure of an estimate to be within a certain percentage of the actual measurement. Over a set of projects, this may be widened to a profile of the proportion of estimates achieving success at a given level q. Conte et al. (1986) define prediction quality P as: P(q) = m / n where m of the n projects have R < q. Again using Kemerer's Function Point data, this may be tabulated or graphed (Figure 1): P(0.25) = 0.33 P(0.5) = 0.47 P(1.0) = 0.60 P(3.0) = 0.86 Figure 1. Cumulative proportion of projects with R < q (iii) DeMarco (1982) employs the relative error to depict the improvement in estimates over the scope of the project. In order to give a positive correlation with the MPMRE, the metric described below is the inverse of DeMarco's original description. Let t 0, t 1,.. t n be a sequence of times of project milestones after the start of the project with t 0 = 0 and t n the final delivery time. If E i is the revised estimate made at time t i, then r i = E i - A / A

3 The Estimating Quality Factor EQF may be defined as: EQF = n-1 r i=0 i ( t i+1 t i ) t n This metric can be visualised as a ratio: the area from the error times the interval for which that estimate was in effect, divided by the area from the actual times the total period (see Figure 2). Perfect estimation would have an EQF of zero. Figure 2. EQF as the ratio of dark shaded area to (A x t 3 ) As an illustration, the fifteen items of Kemerer's data could be considered as the relative error r i of fifteen equally spaced estimates of the same project. This gives EQF = Σ r i ( t i+1 - t i ) / t n = Σ r i (1/15) /1 = the same as the MPMRE. 3. Desirable Properties for Software Metrics Before defining the two new metrics for the meta-measurement of the estimation process, it is appropriate to consider desirable properties for their design. The thirteen qualities for metrics used in this research are an expanded list from those given by Watts (1987) and Kitchenham (1996) and are summarised in Table Measuring convergence Several researchers (for example, Selby, 1991 and Kitchenham, 1996), point out that any estimation model should take advantage of a staged approach, whereby new estimates are made as soon as a better predictor (or more information on the scope or design of the project), becomes available. The most important aspect of a staged process is to incorporate feedback. Selby (1991) asserts: "A fundamental principle is to make measurement active by integrating measurement and process, which contrasts with the primarily passive use of measurement in the past." DeMarco (1982) suggests an alternative EQF by using a weighting of the various estimates made during a project, but interestingly reports that some managers: "feel obliged to stress more heavily the estimates made at the beginning, and don't care terribly much about the last ten to twenty percent convergence at the end". Accordingly, his Time-Weighted EQF is biased towards the initial estimates.

4 T L Woodings 1. Objective Is independent of anyone's opinion 2. Reproducible Can be consistently repeated 3. Standardised Uses a mathematically appropriate scale 4. Valid Is clearly related to the feature being measured - it monotonically increases as the feature rises 5. Precise Is sensitive to changes in the feature measured 6. Robust Is not easily manipulated or sensitive to extraneous factors 7. Comparable Is highly correlated with other metrics measuring the same feature 8. Timely Can be obtained in time for action to be taken on its message 9. Sustainable Is likely to be valid in the future so that trend forecasts based on the metric will be effective 10. Universal Can be translated into sub-metrics for lower parts of the product or process 11. Economical Does not consume significant resources (preferably a byproduct of other activities) 12. Cost-Effective Provides a return on investment 13. Useful/Relevant Supports the goals of the organisation Table One - Desirable properties for a software metric However, elsewhere in his book, DeMarco gives the rule: "Success for the estimator must be defined as a function of convergence of the estimate to the actual, and of nothing else." He goes on to state: "With such incentives, the estimator has no inclination to dodge opportunities for re-estimation. Any change in the right direction will improve the final judgement of estimate quality, and the earlier the change is made the better." A linear weighting based on the time for the centre of each dark shaded area in Figure 2 would appear to be a reasonable candidate for the new metric to reward rapid convergence of E i towards A. That is the weight = ( t i+1 + t i ) / 2 - t 0 where t 0 = 0 as stated earlier.

5 Applying this to the r i in the EQF gives the Convergence of Estimate metric as: CE = n 1 r i=0 i t n ( t i+1 t i)t i+1 + t i n 1 t i+1 + t i 2 i=0( ) ( ) 2 ( ) ( ) n 1 r t 2 2 n i=0 i i+1 t i E i A t i=0 i+1 t i = = n 1 n 1 t n t n + 2 t i=1 i t n At n + 2 t i i=1 Based on DeMarco's comments, a CE of 0.25 should be within the capability of the average software supplier. 5. Assessing a set of estimates The use of such metrics as those above, looks at the accuracy of the estimates after the project is complete and the actual value known. However, there is a need to focus attention on the accuracy of the initial estimate at the start of the project. Conventionally, this is done be requesting the variance or a confidence interval for the estimate - generally obtained directly from a set of expert opinions or by means of the Beta-PERT approach (Malcolm, 1959). Figure 3 shows the distribution of many actual project completion dates standardised about a nominal estimated target date for a typical software organisation. Figure 3. The distribution of actual completion dates It is possible to invert this approach. Instead of a hypothetical distribution of actual completion dates around a fixed estimate, there is a distribution of known independent estimates about a hypothetical actual (that is, unknown at start of project) delivery date. For a given set of estimates, the minimum of the proportion with errors above a certain limit with respect to a variable 'actual' gives a Lower Bound on Estimate Accuracy (LBEA) for the group effort. Although this gives little more information than having the standard deviation of the estimates, it produces a quite different appreciation when labelled "At least x% of the estimators are in error by at least y%". For example, it may be employed to impress upon software engineering students the difficulty of the estimation task. It also gives an indication of convergence for experts using Delphi methods (Boehm, 1981). Suppose, as in section 2 (ii), q is set at 0.25 as a reasonable limit on the relative error R. The LBEA is based on the proportion of estimates which have R > q. That is, A is varied so that the proportion of estimates below (1-q)A plus those above (1+q)A is a minimum (see Figure 4). This is equivalent to maximising the number of estimates within the interval [(1-q)A, (1+q)A].

6 T L Woodings Figure 4. LBEA as a minimum shaded area Let f(t) be the frequency distribution of the estimates over time. Then the proportion of estimates outside the interval is given by: (1+q )A g( A ) = 1 f(t)dt (1 q )A The LBEA is that minimum value of g found when dg/da = 0. A non-parametric equivalent of the LBEA is simple to compute. Given n estimates, rank them in ascending order and select each in turn as the interval lower limit L. The upper limit U = (1+q) L / (1-q). For q = 0.25, U = 1.67L. Thus for each, count the number of estimates m in the interval [L, 1.67L]. Then the LBEA is that value of g = 1 - m/n for which m is greatest. This may be set up without difficulty on a spreadsheet. As an example of the use of the LBEA metric, 20 undergraduates starting a software engineering course were asked to estimate the Spelling Checker given in Fenton (1997). The estimates of effort, in person-days were: {0.8, 2, 3, 5, 6, 10, 10, 13, 20, 20, 20, 20, 60, 80, 100, 100, 100, 160, 160, 180} The mean is 53.5, the standard deviation 60.0, the coefficient of variation 1.12 and the ratio of the largest to smallest 225:1. For a relative error of 0.25, m is computed for each of the twenty estimates: {1, 2, 2, 2, 3, 3, 3, 5, 4, 4, 4, 4, 5, 5, 5, 5, 5, 3, 3, 1} Note that in this case the list is bimodal and thus the function g has two minima The LBEA = 1-5 / 20 = 0.75 which is not surprising given the large range of estimates. That is to say a minimum of 75% of the estimates will be in error by more than 25%. 6. Conclusion Two meta-metrics have been proposed to assess software development parameter estimation and thus focus attention on areas of potential improvement. Both achieve the desirable requirements for metrics. The first, Convergence of Estimate, CE, recognises and rewards the re-estimation of parameters during a project. The second, the Lower Bound on Estimator Accuracy, LBEA, focuses attention on the acceptability of the spread of initial estimates. Research is continuing on the effectiveness of these approaches in improving process estimation in teaching and industry.

7 References Boehm, B W (1981) "Software Engineering Economics", Prentice Hall. Conte, S D, Dunsmore, H E and Shen, V Y (1986) "Software Engineering Metrics and Models", Benjamin-Cummings. DeMarco, T (1982) "Controlling Software Projects", Yourdon Press. Fenton, N E and Pfleeger, S L (1997) "Software Metrics - A Rigorous and Practical Approach" (2nd edition), International Thomson Publishing. Kemerer, C (1987) "An Empirical Validation of Software Cost Estimation Models", Communications of the ACM, 30:5 pp Kitchenham, B A (1996) "Software Metrics - Measurement for Software Process Improvement", NCC-Blackwell. Lederer, A L and Prasad, J (1998) "A Causal Model for Software Cost Estimating Error", IEEE Transactions on Software Engineering, 24:2 pp Malcolm, D G, Roseboom, J H and Clark, C E (1959) "Application of a Technique for Research and Development Program Evaluation", Operations Research, 7 pp Selby, R W, Porter, A A, Schmidt, D C and Berney, J (1991) "Metric-Driven Analysis and Feedback Systems for Enabling Empirically Guided Software Development", Proceedings of the Thirteenth International Conference on Software Engineering, Austin, pp Watts, R A (1987) "Measuring Software Quality", NCC. Woodings, T L (1995) "A Taxonomy of Software Metrics", Software Process Improvement Network (SPIN), available from Comast Consulting, Perth.