Integrating Contract Risk with Schedule and Cost Estimates

Transcription

1

2 Integrating Contract Risk with Schedule and Cost Estimates Breakout Session # B01 Donald E. Shannon, Owner, The Contract Coach December 14, :15pm 3:30pm 1 1

3 The Importance of Estimates Estimates form the basis for budgetary inputs and/or proposal costs and pricing The term accurate estimate is an oxymoron There are various degrees of accuracy with estimates Information costs money how much do you want to know? Faulty estimates lead to Program overruns Failed projects Lack of public or customer confidence Incomplete or delayed completion 2 2

4 What s Wrong with the Current Estimating Process? Who creates the estimates? Individual vs. team Training, certification, experience level and expertise of estimators We assign a single value to what is arguably a range of values We fail to account for variability also know as uncertainty We ignore risk usually because we don t know how to include it Our estimates contain expert opinion Optimistic estimates Advocacy 3 3

5 Efforts to Improve the Process Most improvement efforts have been focused at high-level programs e.g., ACAT I, II, or III GAO Cost Assessment Guide (2012) NASA Analytic Method for Probabilistic Cost and Schedule Risk Analysis (2013) USAF Cost Risk and Uncertainty Analysis Handbook (2007) Techniques discussed in these documents are scalable to smaller efforts. 4 4

6 Recommendations From These Studies The GAO 12-step method provides a useful process for creating meaningful cost estimates Estimates are still difficult to generate because We often lack of meaningful data We face technical uncertainties Project risks Dealing with Project Advocacy Best practice is to generate estimates inclusive of risk and uncertainty Best practice is to express estimates as statistical probabilities / confidence intervals 5 5

7 Building Better Estimates Dealing with the Process Issues 6

8 Sources of Data Actual performance data (cost and work) from identical effort (Historical approach) Built-up engineering estimate (Bottom-up approach) Actual material data Vendor quotes, previous purchases Actual or engineering estimate of labor previous work as baseline Relationship based on functions or characteristics (CER approach) Scaled cost and work from similar effort (Analogy approach) 7 7

9 Purify Data Historical data is preferred May be outdated due to economic factors May contain data from two or more sources Costs attributable to the work Costs attributable to one or more risk events Scrutinize data to remove contaminants 8 8

10 Consider Variability in Performance Time or Cost Estimates Repetitive tasks require more or less time with each performance The task time varies due to unknown complexities Variability = Uncertainty Probabilistic description or time range is commonly used to describe Modeling the Uncertainty of Surgical Procedure Times: Comparison of Log-normal and Normal Models, d%c2%a0p.+strum 9

11 Be Skeptical Concerning Expert Opinion Tendency towards Optimism Bias Underestimate labor hours, cost, and/or duration Predict early availability of technology Most pronounced on estimates of known or frequently performed tasks 1 Counter effects by requesting expert to render 3-point estimate with most likely estimate first 2, then obtain best and worst case. Counter effects with Delphi technique 1. Roy, Michael M. and Christenfeld, Nicholas J. S., Bias in Memory Predicts Bias in Estimation of Future Task Duration Memory & Cognition, 35 (2), p Alleman, G, Why 3 Point Estimates Create False Optimism (Part 1). [Webpage] PM Toolbox, March 17. Available From:

12 Cautions Concerning Expert Opinion Studies have shown that some experts view generating 3-point estimates as onerous, leading to frivolous estimates 1 Project Advocacy: experts tend to dismiss opinions or estimates that would endanger advancement (or continuation) of their program 2 1. Trietsch, D, Mazmanyan, L, Gevorgyan, L and Baker, K, Modeling Activity Times by the Parkinson Distribution with a Lognormal Core: Theory and Validation. [Article] Dartmouth.edu. Available From: scheduing/modelingactivitytimes.pdf Christensen, David S. Ph.D., Project Advocacy and the Estimate at Completion Problem. Journal of Cost Analysis (Spring), p

13 Include Uncertainty in Cost or Schedule Models Expert opinion estimates are best represented by range of values vice a single point estimate Most Likely value defines expected cost or duration based on rules and assumptions as stated. Best Case value represents expected cost or duration if all assumptions stack up on the favorable end of the spectrum Worst Case value represents expected cost or duration should all assumptions stack up on the pessimistic end of the spectrum Most Likely (m) Best Case (a) Worst case (b) Typically Note: Letters the used Most to denote Likely the various value values has (a, less m, b or a,b,c ) vary among researchers so some care should than a 50% opportunity of being correct. be exercised when examining data to determine which The mean letter (μ) is tends associated to with be which closer value. to 50%, consequently the mean is recommended. 12

14 Add Uncertainty to Program Schedules Task durations in project schedules is no longer recorded as a single value We use the PERT data entry screen Each Task has three values Low (Best Base (Most Likely) High (Worst Case) 13

15 Add Uncertainty to Costs Per Occurrence Costs Model as three points based on historical data May model as threepoint estimate based on expert opinion Labor driven costs Assign resources to project schedule Labor costs will automatically vary with scheduled time 14

16 Properly Account for Risk? Risk is a binary event It consist of a condition If this happens Followed by an outcome then this will happen The if is usually a probability there is a 1 in 1000 chance Twelve percent of the time.. The outcome may be A fixed outcome (e.g., start over) Some variable outcome (e.g., a cost, a delay, or both) A probabilistic branch (e.g., implement an engineering change, a recall program, etc.) 15 15

17 Details, Details, Details The hard part about considering variability and risk is quantifying the details How much does an unknown vary? How often does a risk event transpire? How do I quantify risk outcomes? Qualitative Quantitatively 16 16

18 Perform Qualitative Risk Analysis Calculate Risk Score Score = Probability x Impact Record result in Risk Register Plot results on Risk Matrix (optional) 17

19 Assign Values to Risks Expert Opinion to identify likelihood and impact Acceptable method but may be less precise than historical data Must control optimism bias Use Delphi technique if possible (next slide) Obtain consensus Express estimate as value plus/minus a range 18

20 Perform Quantitative Assessment Using mathematic or statistical techniques to: Quantify the likelihood of risk events (Probability) Quantify the impact of risk events Expected Value $$$ Schedule delay Perform sensitivity analysis Risk impact(s) on entire project Risks considered apart from other risks for their unique contribution to overall risk Image from Careercast.com 19

21 Assign Values to Risks Historical Data Preferred method Extract impact data from historical data Cost Schedule delay Litigation Injury Identify frequency of occurrence (i.e., probability) Identify trigger events or precursors Screen Image from Risky Project Intavar Software 20

22 Assign Risks to the Schedule Screen Image from Risky Project Intavar Software 21 21

23 Simulate the Project and Review Results 22 22

24 Simulate the Project and Review Results 23 23

25 Driving to Work An exercise in developing a risk loaded estimate. 24

26 The Driving to Work Problem? You do it every day 12.5 Miles 15 Minutes on a really good day 17 Minutes on a typical day 25 Minutes when traffic is heavy Once and a while there is an accident Delay from 3 to 25 minutes Once you had a flat tire It took 15 minutes to change the tire Three or four times a year you have snow and ice that adds another 7 to 10 minutes What s the likelihood your trip to work on any given day will exceed 22 minutes? 25 25

27 Here s How We Answer That Question Risk a Risk b Probabilistic Duration (or cost) Estimate Risk c The technique used to sum the various distributions can be analytic (e.g., Method of Moments or FRISK) or simulative 1. Grinstead, Charles and Snell, Laurie, Sums of Independent Random Variables. An Introduction to Probability. 2 ed. American Mathematical Society. pg Lurie, Philip M., Goldberg, Mathew, and Robertson, Mitchell, A Handbook of Cost Risk Analysis Methods, IDA Paper P2734, Institute for Defense Analysis, April, 1993pp15 26

28 Solution via Monte Carlo Simulation 27 27

29 Repeat the Single Iteration Numerous Times and Evaluate the Results µ = σ -1 σ 1 σ 2 σ 3 σ 28

30 Use the Underlying Probability Distribution to Make an Estimate and Confidence Interval Almost certainly a lognormal distribution 1 Parameters are µ = σ = Probability of x<22 =.80 Probability of X>22 = A variable might be modeled as log-normal if it can be thought of as the multiplicative product of many independent random variables each of which is positive. P represents the distribution mean (µ) and Q is representative of the distribution standard deviation (σ) in the work from which the formulae were extracted. Modeling Activity Times by the Parkinson Distribution with a Lognormal Core: Theory and Validation 29 29

31 Questions? Let s discuss what we just covered. 30

32 Contact Information Donald E. Shannon Telephone: (505) Web: