It's the mistakes that count!

Transcription

1 It's the mistakes that count! Adrian Cowderoy City University London, England Adrian Cowderoy (Europe) Reading, England This is an account of how to improve existing estimation practices. It is based on the empirical research results of the MERMAID project. (ESPRIT project 2046, partially-funded by the European Commission from 1988 to 1992.) It also includes subsequent practical experience of using and teachg estimation practices. Copyright (c) Adrian C:owderoy 1996

2 Why was the estimate wrong? Estimate accuracy Estimate range and confidence level Estimate convergence Estimation method An estimate is a prediction of an attribute value that can not yet be measured, but which will be. (If there is no intention to compare the actual outcome to the estimate then the process can not be called "estimation".) The estimate "range" consists of a high and low value beyond which it is tmlikeldv that the actual value will go. - Statisticians know that- a percentage of estimates will be outside this range (hence the confidence limit). - Project managers know that there are also many other uncertainties in the project and so it is easier to treat the range as "correct" until there is evidence otherwise. - Estimator's know that the estimate range can be even more crazy than the estimate itself. When a project is first conceived, the estimates of effort and duration tend to be wild. As detail about the project increases the estimates become better. When the project starts and initial size information is available the estimates become even better. There may be periods of gross pessimism or optimism in estimation (such as during testing) but eventually the estimate of how many days or hours are left will be accurate. This general trend is "estimate convergence". It is common practice to blame the estimation method when an estimate is later found to be wrong. ("A good worker always blames hls tools".) More typically the faults are likely to be in the materials obtained by the worker (i.e. the information about the project). Or the worker has misused the tools.

3 Why was the estimate wrong? Assumptions Faults in the method Project data "it was bigger than we expected" Current project is exceptional calculation or data entry fault Faults in the estimation method (or tools) do cause errors. The accuracy of these is often published ("the model is accurate for 80% of projects to within 2.5%"). This typically refers to analysis of past data. It does not mean the new estimate will be accurate because estimation methods are not the only source of error: (1) The data about the current project is frequently vague at the earlier phases of the project where the estimates are performed. Some attributes may not even be known, such as the calibre of the staff that will be allocated. (2) There is often a claim that the current project is an exception. If this were true it could be a significant source of error in the estimate. However the statement is frequently not true when the estimati'on is based on a model. Such models represent the full variety of projects on which they were calibrated, of whlch at least some may have been similar to the current one. The new project does not have to be typical of an "average" project. (3) One of the most common sources of error in estimates is the type that should be easiest to avoid: calculation errors (such as incorrectly entered formulae in spreadsheets) and data entry errors (into tolols).

4 Why was the estimate wrong? Unexpected problems. Potential problems... compared to last time (included). Potential disasters (excluded)... use what-if scenarios... state as an explicit assumption There is relationship between estimation and the existence of risk in a project. All projects have risk. If there was no risk the work would be produced using factory methods rather than as a project. That is, the new project will encounter problems. The question is, will the total number and size of these problems be unusual? If the risk was unusual for this type of project then the estimate might be adjusted accordingly - but only if the risk is unusual. The risk within a project consists entirely of potential disasters and potential problems. The problems are the frequent occurrencesand the disasters have (hopefully) very low probability of occurrence. The disasters can cause the project to fail or to be serious& restructured. When that happens the original estimate will be wrong. For each potential disaster the size of that risk should be estimated as a what-if scenario. The "main" estimate can then have the explicit assumption that none of the disasters will happen.

5 What could go wrong this time? Assumption identification Method faults Project data... KLOC, CMPLX, PCAP, etc. Mismatch Potential disasters + calculation or data errors When estimating, consider the assumptions underlying the estimate. That is, what could go wrong? As a first step, tist all the assumptions. The next step will be to establish whether they deserve further concern. What matters fust is to ensure that nothing isforgotten. There are four major assumptions within an estimate: - that there are no.faults in the method - that the data used in the calculations is perfect - that the current project is not an exception - that there will be no disasters in the project (There is also the assumption that the calculations are performed without clerical error.) Some of these assumptions are themselves made up of a series of smaller assumptions. For example, the assumption that the data is perfect consists of an assumption that the size is known perfectly, and also the complexity, the proparnrner capability, and so on.

6 How bad could it be? Assumption sizes Excellent 1% Good 2% Reasonable 5% Poor 10% Bad + 20% Terrible k 50% Not known k? For each of the big assumptions, rank their size using a simple scale, such as above. (Different scales are used in different environments.) If the assumption is bad or temble consider breaking the assumption into smaller assumptions that can be studied separately. If the assumption size is "not known" it implies the estimate could be hopelessly wrong. The estimate must either be improved or abandoned.

7 Example Estimate reporting "A good estimate is one that is trusted" Report the largest assumptions and the number of assumptions Demonstrate estimator's past estimating accuracy Estimator's who provide estimates to project managers are providing a service. Likewise project planners who make estimates as part of their work are providing the same service. There are quality attributes to this estimation service. For example, - the accuracy of the nominal estimate, - the reliability of the estimate range, - the clarity of presentation of the estimation, - the timeliness with which the estimate was produced, - the cost of producing the estimate, and - the believabili'ty of the estimate The last of these factors is often the most critical - an estimate that is not believed is likely to be ignored. There are two ways of indicating whether an estimate can be trusted. - List the largest assumptions in the estimate. (Also indicate how many assumptions there are.) Other people can frequently understand assumptions and they may have ideas of how to improve them. - Provide evidence that the estimator has a good track record. (Hence it is worthwhile to monitor your own estimates and keep a list of successes and failures, and how the failures could have been avoided.)

8 pppppppppp Example Worst-case scenario MM = 2.5 KLOC 1.05 Assumptions (example values): method accuracy + 30% data accuracy + 50% development mode + 0% mismatch + 10% worst case 1.90 MM best case MM / As an example, consider a simple cost model There is an assumption that the method is accurate. Use the model on data fiom past projects to find the accuracy of the assumption. In the example above, perhaps *30%. There is an assumption that the size (KLOC) is known accurately. However if the estimate is being made near the start of the project the KLOC will be a guesstimate. It could easily be wrong by 50%. The development mode can probably be assessed accurately. However is the current project an exception? For example, was the estimation model calibrated on projects from a different company and business to your own? The accuracy of this assumption could be bad or possibly so vague that it can not be quantified - and if it can not be qilantified then do not trust the estimate There is also the assumption that the equation or data are correct - these should be checked. Also there may be potential disasters. In the example here the total size of the assumptions indicates that the actual outcome may be 90% higher than the nominal estimate. (An optimist would suggest that the actual might be about half the estimate.) Such a value represents the extreme ofall the assumptions being wrong. This is unlikely, however the extreme is easy to interpret if quoted with the number of assumptions. (Some pessimistic project managers tend to regard the extreme worst-case as more the realistic than nominal estimate.)

9 Example Analogy estimates MMA = MMB. (LOCA / LOCB). "personnel adjustment" Assumptions: size of current project 220% size of previous project +5% method for adjusting size I % personnel in-current project &20% personnel in analogous project - +I 0% method for adjusting "personnel" k50% differences that have been ignored 220% overlap ko% Analogy estimates are based on comparing the current bit of work with a previous one. (Often this consists of dividing the project into components for which sensible comparisons can be made.) The start-point is to imply that the new work is the same as the analogy. Then the differences between them are considered The-difference ii size is easiiy.considered by comparing the size of the new component to the size of the analogy. The trouble with this adjustment is that the size of the new component may be a guess, and the size of the analogy may be only a memory, and the method of adjusting for the difference in size may not consider the effect of diseconomies (or economies) of scale. Often the differences between the two projects are not made using mathematical rules. Instead there are statements like "the new project has a better team so the productivity will be. better". This includes assumptions that the new team is understood, that memories of the old team are accurate, and that yo11 know how to adjust for the differences in ability. Collectively these assumptions may be so far wrong that it would create less error in the estimate if the personnel differences between the two projects were ignored

10 Example Regression-based estimation y = a + b.x + c.y + d.z +... y = a.xb.yc.zd-- linear non-linear Principle: 1. collect data for local projects (1 0 points?) 2, always measure at the same milestone 3. (simple data is often the best) 4. use regression to find best size driver 5. assess closeness of fit, PRED(25%), MRE, ~2 6. use regression to find next best adjustment does it improve the closeness of fit? 7. keep going until closeness of fit diverges Statistically-based estimates are based on methods such as statistical regression. The easiest equation is y=a-bx, although the same principle can be used for calibrating more complex versions of this equation. The most useful way of using regression is to pick local projects and data that is easily measured early in the lifecycle. For example, the number of menu commands and the number relationships in the (draft) entityrelationship model. Compare l~ke with like, for example using only the data that is available at the end of the specification phase. With regression analysis an indicator is needed of the quality of the fit or, conversely, the size of the error. Unfortunately the value of R* does not fidly indicate the quality of the fit. A combination is need of ~ 2 the, proportion of projects that the equation predicts within 25% (or some similar number), and the mean random error. Use this combination of methods for measuring the error to find which of the items of data provides the single best prediction of size. That becomes the size dnver and gives the best equation. Using a spreadsheet then try including a second item of data as an extra term in the equation. Try each other item of data to see which improves the error most. (If none of them reduce the error then stop.) It is possible to continue and add more terms to the equation, but typically the extra terms increase the error rather than reduce it. Also the statistics are more tricky.

11 Conclusion Errors: "Focus on the big ones" If you want to improve your current estimate, focus yoiu. attention on the biggest assumptions. If you want other people to believe your estimate, focus their attention on the biggest assumptions. If you want to learn how to estimate better in future, study your previous estimates to see which of the assumptions you made were subsequently shown to be wrong.