BAE Systems Risk Opportunity & Uncertainty Modelling ACostE North West Region 4th September 2013

Transcription

1 BAE Systems Risk Opportunity & Uncertainty Modelling ACostE North West Region 4th September 2013 BAE SYSTEMS PLC 2011 All Rights Reserved The copyright in this document, which contains information of a proprietary nature, is vested in BAE SYSTEMS Public Limited Company. The contents of this document may not be used for purposes other than that for which it has been supplied and may not be reproduced, either wholly or in part, in any way whatsoever, nor may it be used by, or it contents divulged to, any person whatsoever without the prior written permission of BAE SYSTEMS Public Limited Company 10/09/2013 1

2 Agenda Part 1 Risk Opportunity and Uncertainty Definitions Brief Overview of Monte Carlo Analysis Shortfalls in Monte Carlo Part 2 Cost & Schedule Integration An Overview of the Slipping and Sliding Technique Summary 2

3 Risk Opportunity & Uncertainty - Definitions Variability within a project cost or schedule can be summarised as either: Uncertainty Something that will happen but the exact values/parameters are not known E.g. Normal car journey from home to work Risk or Opportunity Things which may or may not happen Risk An event or a series of events which, on occurring, would damage a project or business objective in terms of performance, functionality, time of delivery, customer acceptance, or cost. E.g. An accident on the route ahead causes a hold-up on the journey Opportunity An event or series of events which, on occurring, would offer benefit to the project or business in terms of performance, functionality, time of delivery, customer acceptance, or cost E.g. An accident behind us reduces congestion on the journey ahead 3

4 ROU Bottom Up Evaluation In order to do a Bottom-Up assessment of Risk Opportunity and Uncertainty (ROU), it is generally necessary to: Define a work package in an appropriate level of detail (tasks) Assign a range of likely cost and/or schedule outcomes for each task Link tasks that have an underlying relationship in terms of outcome in terms of cost (or time) e.g. Design overrun leads to increasing cost of Manufacture or Construction i.e. Partial Correlation often overlooked in Bottom-up ROU because it is not properly understood Add the tasks together Review the range of outcomes (Failure to correlate tasks appropriately will result in too narrow an output range) Make recommendations based on the range of outcomes possible This approach generally requires the use of a Monte Carlo Simulation toolset 4

5 Monte Carlo Simulation What is it? A Structured Approach to a Chance Encounter! 5

6 Monte Carlo Simulation What is it? Why do we use it? A method of aggregating multiple distributions from independent variables in a manner which maintains statistical correctness Usually not practical to combine distributions manually: Total Probability =1 Opt = 1 Pess = 6 Total Probability =1 Opt = 1 Pess = 6 Total Probability =1 Opt = 2 Pess = 12 6

7 Occurrences Monte Carlo Simulation Why is that wrong? Consider throwing two conventional dice. Each has a uniform distribution. Add together the scores Die What is it doing? Looking at every possible combination Not feasible in complex multivariable environments Score 7

8 Monte Carlo Simulation Combining distributions mathematically is theoretical possible but extremely complex and thus impractical in real terms For every conceivable permutation we want the product of those distributions not the sum of them Monte Carlo Simulation provides an approximation shortcut to that product S f(x) S P f(x) 8

9 Monte Carlo Simulation Monte Carlo Simulation is a statistically valid way of adding together a number of distributions Possible outcomes are defined by the user selecting an appropriate distribution and probability of occurrence for each task Monte Carlo Simulation picks a value randomly from within the range of possible outcomes defined The randomly generated outcomes from all the tasks can be combined together usually by simply adding them together to create a single valid potential outcome of the overall total (sometimes called a slice) This process is repeated many times to generate a distribution of possible outcomes weighted according to the distributions chosen for each task The output distribution generated allows the user to interpret data using confidence levels to define three-point estimates 9

10 Monte Carlo Simulation Occurrences Cost Probability 10,000 Slices Slice 1 Slice 2 Slice 3 20 Slice view ,000 Slice view Cost 10

11 Likelihood of Occurrence Likelihood of Occurrence Likelihood of Occurrence ROU Bottom Up Evaluation Uncertainty Only Model S Optimistic S Pessimistic 0 Potential Outcomes Cost or Time Baseline Task Baseline Uncertainty Risk and Opportunity Only Model Baseline Task + Uncertainty 0 Potential Outcomes Cost or Time Risk Register Based Risk Opportunity and Uncertainty Combined Model 0 Baseline Task Potential Outcomes Baseline Uncertainty & Risk Cost or Time 11

12 Monte Carlo Simulation Things to do to give Monte Carlo Simulation a chance of being right Try to ensure that the distributions you choose are appropriate At least make sure that the basic distribution shape and range are right The majority of inputs to Schedule and Cost Monte Carlo Models are likely to be positively skewed Understand the difference between the three principle Measures of Central Tendency: Mean (Average), Mode (Most Likely)and Median (50% Confidence) Do you mean Minimum and Maximum or Optimistic and Pessimistic? Ask the question about the Mode (Most Likely): In what circumstances... can the value be less than the Most Likely?... can the vale be more than the Most Likely? Apply Correlation to tasks Very few tasks are totally independent of all others Consider a background correlation of between 20-30% (potentially even more for Concept Development) Very few tasks are negatively correlated with others 12

13 Likelihood of Occurrence Likelihood of Occurrence Monte Carlo Simulation: Effect of Correlation Take any two variables in a Monte Carlo Simulation: The impact of positive correlation is to push down and out The impact of negative correlation is to push in and up High values of one variable are associated with high values of another variable and vice versa High values of one variable are associated with low values of another variable and vice versa Range of Potential Outcomes Range of Potential Outcomes 13

14 Monte Carlo Simulation: The Downside Bottom-up Uncertainty Assessment Undefined or Unclear Task Definition Emergent Baseline Task Performance There are known knowns. To These know are that things we we know know what we that know, we know. Undefined and that Risks we do not know what we do not know, that There is are true known knowledge. unknowns. or the That is to say, there are Confucius things Unknown that we Unknowns now know we don t know. Gap in Monte Carlo Analysis Clearly Defined Baseline Estimate Defined Baseline Tasks Will Occur But there are also unknown unknowns. These are things we do not know Risk & we don t know. Opportunity Register Donald Rumsfeld United States Secretary of Defense Likelihood of Task Occurrence May or May Not Occur Bottom-up Risk & Opportunity Assessment 14

15 Cost & Schedule Integration Things that Dreams are made of At the risk of being controversial, perhaps we should start by setting the cat amongst the pigeons. Note: No real pigeons have been harmed in the making of this presentation 15

16 Cost Analysis Cost & Schedule Integration Stuff that Dreams are made of Schedule Analysis Let s be honest if it was easy we d all be doing it, but we re not conclusion: it isn t easy Integrated Cost & Schedule Analysis Schedule elements may require correlation Cost elements may require correlation Both may require cross-correlation Do we even understand it? We will all have heard the old saying Time is Money, and there is some truth in that So we should be looking at Integrated Cost and Schedule Analysis why aren t we? It needs investment: Can we afford to? Can we afford not to? 16

17 So, how can we fix it? In line with good estimating practice, we advocate using more than one approach, method and/or technique to evaluate the potential range of cost outcomes Top-down approach to pick up the unknowns and avoid duplication Takes a view of the schedule risk and pro-rata the rate of spend affected (Time costs money, resources are unlikely to be re-deployed) Commercial/Financial uplift factors (escalation etc) Bottom-up approach based on authorised Risk Registers Use of Monte Carlo Simulation - statistical technique to allow multiple variables and probabilities to be modelled Balanced view of Top-down and Bottom-up approaches By its very nature the Top-down approach ( worst-case view) should be greater than Bottom-up approach If not, or if there is a significant difference either way, it could indicate that either one or other approach is too immature, or overly optimistic or pessimistic 17

18 Cost Example of a Top-down Approach What is it? Schedule Risk Resources/Cost can be scaled in direct proportional to the increase in programme duration This implies a standing army effect if risks materialise Approach can be justified through a number of general principles: Baseline Task Top-Down Variability Resources utilised by a proposal are scoped on the baseline programme. Task Schedule Resources are generally internally re-deployed within a contract Task Task Risks generally manifest themselves as programme slippage Uplift Factors Task Task Task Task These cover issues not sufficiently covered by the schedule risk assessment Task Task Examples include Escalation 18

19 Using Monte Carlo to make a Price Recommendation 100% 80% 60% Our bottom-up approach to estimating looks at modelling risk, opportunity and uncertainty around the Most Likely values 40% 20% 0% We always look to price based on values which have a higher level of Confidence Min: Max: Probability: 100% Sensible Level of Confidence for Bid Audience Quiz: Why do we do that? Min: Max: Probability: 100% We don t trust our own estimates? We are so totally risk-averse? We know that our customers will knock us down? The schedule may slip? The sum of the Most Likely Values is not the most likely value? Monte Carlo Output is misleading? 19

20 Making A Recommendation - Interim For internal clearance, generally prior to price formulation and submission, we provide a high level summary of the cost build up: Bottom-up Functional Budget Request Risk Modelling 50% Confidence level Balancing Number for Total Confidence Level Million Confidence Optimistic Range % Pessimistic Range % Initial Recommendation % Cost Build Up Million % of Total Baseline Budget Request % Technical Risk Contingency % Management Contingency % Total % maybe we could do better Whatever happened to that Top-down Assessment we made? 20

21 Likelihood of Occurrence Taking a Balanced View Top-Down Evaluation Baseline Task Top-Down Variability Schedule? Cost Bottom-Up Confidence Level S Optimistic Opportunities and Uncertainties S Pessimistic Risks and Uncertainties 0 Potential Outcomes Cost Baseline Task Bottom-Up Evaluation Baseline Uncertainty Risk & Opportunity Register Based 21

22 Slipping & Sliding - A Pragmatic Aid to Judgement What is it? Slipping & Sliding is a simple technique we developed on the fly in support of an Estimating Practitioner Training Course on Risk Opportunity & Uncertainty Trainees wanted more guidance on making that Judgement Call And thus, a training exercise was born that we called Slipping & Sliding for reasons that will become abundantly clear The rational is: We assume that we have made an honest and reasonable assessment of the Top-down approach to cost variability based on a conservative view of the schedule risk (higher Confidence Level) We know that the approach is inherently pessimistic We assume that we have made an honest and reasonable assessment of the Bottom-up approach to Risk, Opportunity and Uncertainty using Monte Carlo We know that it does not include any explicit provision for the Rumsfeld Factor, Unknown Unknowns The approach is inherently optimistic Reality lies somewhere between the optimistic and pessimistic view of life 22

23 Slipping & Sliding - A Pragmatic Aid to Judgement Bottom-up Risk & Opportunity First Pass: 70% Confidence on Uncertainty 50% Confidence on Risk & Opportunity Functional Requests Baseline Programme Top-down Assessment (80%SRA) So, in this particular case: 70% Confidence Level + 50% Confidence Level Risk Opportunity & 60% Confidence level which all fits nicely inside the Top-down assessment, which is inherently pessimistic Bottom-up Uncertainty Bottom-up Risk, Opportunity & Uncertainty (for statistical validity)

24 Slipping & Sliding - A Pragmatic Aid to Judgement Bottom-up Risk & Opportunity Top-down Assessment (80%SRA) Functional Requests Baseline Programme Bottom-up Uncertainty So, in this particular case: Our Bottom-up Risk Opportunity & 60% Confidence level May only be equivalent to around the 40% Confidence Level of our Top-down Approach Bottom-up Risk, Opportunity & Uncertainty (for statistical validity)

25 Slipping & Sliding - A Pragmatic Aid to Judgement Bottom-up Risk & Opportunity Top-down Assessment (80%SRA) Functional Requests Baseline Programme Bottom-up Uncertainty whereas our Top-down 50% Confidence level may be equivalent to around the 72% Confidence Level on our Bottom-up Approach Bottom-up Risk, Opportunity & Uncertainty (for statistical validity)

26 Slipping & Sliding - A Pragmatic Aid to Judgement Bottom-up Risk & Opportunity Functional Requests Baseline Programme Bottom-up Uncertainty Top-down Assessment (80%SRA) } Adjustment for inherent Optimism Bias in the bottom-up approach Bottom-up Risk, Opportunity & Uncertainty (for statistical validity) Case 1 Ratio of Top-down to Bottom-up Approaches: Greater than but close to one Conclusion? The two approaches are reasonably consistent with one another

27 Slipping & Sliding - A Pragmatic Aid to Judgement Bottom-up Risk & Opportunity Top-down Assessment (80%SRA) Case Ratio of Top-down to Bottom-up Approaches: Significantly greater than one Functional Requests Baseline Programme Bottom-up Uncertainty Conclusion? The two approaches are not consistent with one another, either: the top-down is overly pessimistic or, the bottom-up is too immature Bottom-up Risk, Opportunity & Uncertainty (for statistical validity)

28 Slipping & Sliding - A Pragmatic Aid to Judgement Bottom-up Risk & Opportunity Top-down Assessment (80%SRA) Case 3 Ratio of Top-down to Bottom-up Approaches: Less than one Functional Requests Baseline Programme Bottom-up Uncertainty Conclusion? The two approaches are not consistent with one another an optimistic value cannot be greater than a pessimistic value. Either: the top-down is overly optimistic or, the bottom-up is too immature Bottom-up Risk, Opportunity & Uncertainty (for statistical validity)

29 Making A Recommendation - Revisited Following consideration of the inherent Optimism Bias in the Bottom-up Approach, we can modify our high level summary of the cost build up: Bottom-up Functional Request Risk Modelling 50% Confidence level Balancing Number for Total Confidence Level Million Confidence Optimistic Range % Pessimistic Range % Revised Recommendation % Cost Build Up Million % of Total Baseline Budget Request % Technical Risk Contingency % Management Contingency % Total % the inherent adjustment is held within Management Reserve or a higher Technical Risk Contingency 29

30 Slipping & Sliding - A Pragmatic Aid to Judgement Does it work in Practice? We do not claim it to be a perfect substitute for holistic Cost and Schedule Integration The technique is not a perfect solution but none are We re looking for reasonable accuracy not unreasonable precision It does not replace estimating judgement it can guide the thought process in making a judgement To narrow the range or eliminate the extremes To reject or rework a particular approach It provides a degree of Quality Control in our approach to generating three-point estimates Proof of the Pudding as they say is in the eating Let s look at an example 30

31 Cost Including Risk ( K) Example Comparison Based on Real Data Integrated Cost and Schedule 30,000 25,000 20,000 15,000 10,000 5, Duration (Months) 31

32 Cost Including Risk ( K) Example Comparison Based on Real Data Integrated Cost and Schedule 31,000 30,000 29,000 28,000 27,000 26,000 25,000 24, Duration (Months)

33 Cost Including Risk ( K) Example Comparison Based on Real Data Integrated Cost and Schedule 31,000 30,000 29,000 28,000 27,000 26,000 25,000 24, Duration (Months) 33

34 Slipping & Sliding - A Pragmatic Judgement Aide Confidence Level of the Final Agreement a Reality Check Confidence levels and intervals are calculated as outputs from the Monte Carlo modelling exercise. However, it should be remembered that the Confidence Level generated through Monte Carlo Analysis is overstated Reality is that the true value at any Confidence Level above the Mode will be greater than calculated, because The value returned by Monte Carlo is only based on the analysis of the inputs It excludes any consideration of things that have not been considered Unknown Unknowns It assumes the baseline programme Pessimism Bias Failure to correlate tasks will narrow the output range too much The Top-down Method is inherently pessimistic in nature It assumes that if the one task slips, everything slips - there is no recovery However, it does create headroom for those unknown unknowns Reality is likely to be somewhere between the two views Optimism Bias 34

35 Slipping & Sliding - A Pragmatic Judgement Aide Thank you for listening Any questions? 35