BAE Systems SCAF Presentation June BAE SYSTEMS 2013, all rights reserved Unclassified 31/07/2013 1

Transcription

1 BAE Systems SCAF Presentation June 2013 BAE SYSTEMS 2013, all rights reserved Unclassified 31/07/2013 1

2 Agenda An Alternative Approach to Cost and Schedule Integration BAE Systems Commercial Estimating Approach to Risk Opportunity and Uncertainty Evaluation An Overview of the Slipping and Sliding Technique A Real-Life Example 2

3 Cost & Schedule Integration Things that Dreams are made of At the risk of being controversial, perhaps we should start by setting the cat amongst the pigeons. Note: No real pigeons have been harmed in the making of this presentation 3

4 Cost Analysis Cost & Schedule Integration Stuff that Dreams are made of Schedule Analysis Let s be honest if it was easy we d all be doing it, but we re not conclusion: it isn t easy Integrated Cost & Schedule Analysis Schedule elements may require correlation Cost elements may require correlation Both may require cross-correlation Do we even understand it? We will all have heard the old saying Time is Money, and there is some truth in that So we should be looking at Integrated Cost and Schedule Analysis why aren t we? It needs investment: Can we afford to? Can we afford not to? 4

5 Is there a simpler alternative? In line with good estimating practice, we advocate using more than one approach, method and/or technique to evaluate the potential range of cost outcomes Top-down approach to pick up the unknowns and avoid duplication Takes a view of the schedule risk and pro-rata the rate of spend affected (Time costs money, resources are unlikely to be re-deployed) Commercial/Financial uplift factors (escalation etc) Bottom-up approach based on authorised Risk Registers Use of Monte Carlo Simulation - statistical technique to allow multiple variables and probabilities to be modelled Balanced view of Top-down and Bottom-up approaches By its very nature the Top-down approach ( worst-case view) should be greater than Bottom-up approach If not, or if there is a significant difference either way, it could indicate that either one or other approach is too immature, or overly optimistic or pessimistic 5

6 Cost Top-down Approach What is it? Schedule Risk Resources/Cost can be scaled in direct proportional to the increase in programme duration This implies a standing army effect if risks materialise Approach can be justified through a number of general principles: Baseline Task Top-Down Variability Resources utilised by a proposal are scoped on the baseline programme. Task Schedule Resources are generally internally re-deployed within a contract Task Task Risks generally manifest themselves as programme slippage Uplift Factors Task Task Task Task These cover issues not sufficiently covered by the schedule risk assessment Task Task Examples include Escalation 6

7 Using Monte Carlo to make a Price Recommendation 100% 80% 60% Our bottom-up approach to estimating looks at modelling risk, opportunity and uncertainty around the Most Likely values 40% 20% 0% We always look to price based on values which have a higher level of Confidence Min: Max: Probability: 100% Sensible Level of Confidence for Bid Audience Quiz: Why do we do that? Min: Max: Probability: 100% We don t trust our own estimates? We are so totally risk-averse? We know that our customers will knock us down? The schedule may slip? The sum of the Most Likely Values are not the most likely values? Monte Carlo Output is misleading? 7

8 Generating an Estimate Recommendation Bottom-up Uncertainty Assessment Undefined or Unclear Task Definition Emergent Baseline Task Performance There are known knowns. To These know are that things we we know know what we that know, we know. Undefined and that Risks we do not know what we do not know, that There is are true known knowledge. unknowns. or the That is to say, there are Confucius things Unknown that we Unknowns now know we don t know. Top-down Reality Check Clearly Defined Baseline Estimate Defined Baseline Tasks Will Occur But there are also unknown unknowns. These are things we do not know Risk & we don t know. Opportunity Register Donald Rumsfeld United States Secretary of Defense Likelihood of Task Occurrence May or May Not Occur Bottom-up Risk & Opportunity Assessment 8

9 Making A Recommendation - Interim For internal clearance, generally prior to price formulation and submission, we provide a high level summary of the cost build up: Confidence Level Million Confidence Optimistic Range % Pessimistic Range % Recommendation % Bottom-up Functional Request Risk Modelling 50% Confidence level Cost Build Up Million % of Total ERF Returns % Technical Risk Contingency % Management Contingency % Balancing Number for Total Total % maybe we could do better Whatever happened to that Top-down Assessment we made? 9

10 Likelihood of Occurrence Taking a Balanced View Top-Down Evaluation Baseline Task Top-Down Variability Schedule? Cost Bottom-Up Confidence Level S Optimistic Opportunities and Uncertainties S Pessimistic Risks and Uncertainties 0 Potential Outcomes Cost Baseline Task Bottom-Up Evaluation Baseline Uncertainty Risk & Opportunity Register Based 10

11 Slipping & Sliding - A Pragmatic Judgement Aide Confidence Level of the Final Agreement a Reality Check Confidence levels and intervals are calculated as outputs from the Monte Carlo modelling exercise. However, it should be remembered that the Confidence Level generated through Monte Carlo Analysis is overstated Reality is that the true value at any Confidence Level above the Mode will be greater than calculated, because The value returned by Monte Carlo is only based on the analysis of the inputs It excludes any consideration of things that have not been considered Unknown Unknowns Pessimism Bias It assumes the baseline programme The Top-down Method is inherently pessimistic in nature It assumes that if the one task slips, everything slips - there is no recovery However, it does create headroom for those unknown unknowns Reality is likely to be somewhere between the two views Optimism Bias 11

12 Slipping & Sliding - A Pragmatic Aid to Judgement What is it? Slipping & Sliding is a simple technique we developed on the fly in support of an Estimating Practitioner Training Course on Risk Opportunity & Uncertainty Trainees wanted more guidance on making that Judgement Call And thus, a training exercise was born that we called Slipping & Sliding for reasons that will become abundantly clear The rational is: We assume that we have made an honest and reasonable assessment of the Top-down approach to cost variability based on a conservative view of the schedule risk (higher Confidence Level) We know that the approach is inherently pessimistic We assume that we have made an honest and reasonable assessment of the Bottom-up approach to Risk, Opportunity and Uncertainty using Monte Carlo We know that it does not include any explicit provision for the Rumsfeld Factor, Unknown Unknowns The approach is inherently optimistic Reality lies somewhere between the optimistic and pessimistic view of life 12

13 Slipping & Sliding - A Pragmatic Aid to Judgement Bottom-up Risk & Opportunity First Pass: 70% Confidence on Uncertainty 50% Confidence on Risk & Opportunity Functional Requests Baseline Programme Top-down Assessment (80%SRA) So, in this particular case: 70% Confidence Level + 50% Confidence Level Risk Opportunity & 60% Confidence level which all fits nicely inside the Top-down assessment, which is inherently pessimistic Bottom-up Uncertainty Bottom-up Risk, Opportunity & Uncertainty (for statistical validity)

14 Slipping & Sliding - A Pragmatic Aid to Judgement Bottom-up Risk & Opportunity Top-down Assessment (80%SRA) Functional Requests Baseline Programme Bottom-up Uncertainty So, in this particular case: Our Bottom-up Risk Opportunity & 60% Confidence level May only be equivalent to around the 40% Confidence Level of our Top-down Approach Bottom-up Risk, Opportunity & Uncertainty (for statistical validity)

15 Slipping & Sliding - A Pragmatic Aid to Judgement Bottom-up Risk & Opportunity Top-down Assessment (80%SRA) Functional Requests Baseline Programme Bottom-up Uncertainty whereas our Top-down 50% Confidence level may be equivalent to around the 72% Confidence Level on our Bottom-up Approach Bottom-up Risk, Opportunity & Uncertainty (for statistical validity)

16 Slipping & Sliding - A Pragmatic Aid to Judgement Bottom-up Risk & Opportunity Functional Requests Baseline Programme Bottom-up Uncertainty Top-down Assessment (80%SRA) } Adjustment for inherent Optimism Bias in the bottom-up approach Bottom-up Risk, Opportunity & Uncertainty (for statistical validity) Case 1 Ratio of Top-down to Bottom-up Approaches: Greater than but close to one Conclusion? The two approaches are reasonably consistent with one another

17 Slipping & Sliding - A Pragmatic Aid to Judgement Bottom-up Risk & Opportunity Top-down Assessment (80%SRA) Case Ratio of Top-down to Bottom-up Approaches: Significantly greater than one Functional Requests Baseline Programme Bottom-up Uncertainty Conclusion? The two approaches are not consistent with one another, either: the top-down is overly pessimistic or, the bottom-up is too immature Bottom-up Risk, Opportunity & Uncertainty (for statistical validity)

18 Slipping & Sliding - A Pragmatic Aid to Judgement Bottom-up Risk & Opportunity Top-down Assessment (80%SRA) Case 3 Ratio of Top-down to Bottom-up Approaches: Less than one Functional Requests Baseline Programme Bottom-up Uncertainty Conclusion? The two approaches are not consistent with one another an optimistic value cannot be greater than a pessimistic value. Either: the top-down is overly optimistic or, the bottom-up is too immature Bottom-up Risk, Opportunity & Uncertainty (for statistical validity)

19 Making A Recommendation - Revisited Following consideration of the inherent Optimism Bias in the Bottom-up Approach, we can modify our high level summary of the cost build up: Bottom-up Functional Request Risk Modelling 50% Confidence level Balancing Number for Total Confidence Level Million Confidence Optimistic Range % Pessimistic Range % Recommendation % Cost Build Up Million % of Total ERF Returns % Technical Risk Contingency % Management Contingency % Total % the inherent adjustment is held within Management Reserve or a higher Technical Risk Contingency 19

20 Slipping & Sliding - A Pragmatic Aid to Judgement Does it work in Practice? We do not claim it to be a perfect substitute for holistic Cost and Schedule Integration The technique is not a perfect solution but none are We re looking for reasonable accuracy not unreasonable precision It does not replace estimating judgement it can guide the thought process in making a judgement To narrow the range or eliminate the extremes To reject or rework a particular approach It provides a degree of Quality Control in our approach to generating three-point estimates Proof of the Pudding as they say is in the eating Let s look at an example 20

21 21

22 Example Based on Real Data Salient Details Small to Medium sized upgrade to a Military fast-jet Navigation System Programme covers: Design Development Test & Integration Upgrade to 47 Aircraft on a Return To Works programme Initial Support Baseline duration 70 months Baseline Cost (excluding risk and other contingencies) million 5 Key Risks mainly covering clearance aspects of the upgrade in particular integration with other systems 22

23 Example Based on Real Data Schedule Analysis - Details Total Baseline Duration 70 months Design 24 months Development 21 months Test & Integration 24 months Upgrade to 47 Aircraft 28 months Initial Support 25 months Significant concurrency over Design Development and Test & Integration 67 activities referenced 20 key schedule drivers identified 3-Point Estimates provided for the duration of all key schedule drivers 5 risks identified 3-Point Duration Estimates provided for the all risks 24 logical links made between activities 23

24 Example Based on Real Data Schedule Analysis - Level 1 Bar Chart Programme Year Month D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S Months From ITP Design Develpoment Test & Integration Upgrade to 47 Aircraft Initial Support Using the dataset generated a Monte Carlo Analysis of the schedule was undertaken 24

25 Confidence Levels Example Based on Real Data Schedule Analysis - Monte Carlo Output % Confidence Level % increase in programme duration 12.26% Confidence Level Months Programme Duration (months) 25

26 Example Based on Real Data Schedule Analysis - Monte Carlo Output Deterministic Duration of 70 months has only a 12.26% confidence level of being achieved Reference confidence Levels: 10% Confidence Level months 50% Confidence Level months 90% Confidence Level months Additional Point of Interest 80% Confidence Level months Duration increase of 14.5% against the Baseline Programme 26

27 Example Based on Real Data Cost Analysis - Details Baseline Cost (excluding risk and other contingencies) Programme Management million Design million Development Test & Integration million Purchases million Production Upgrade to 47 Aircraft million Initial Support million Grand Total million 5 risks identified 3-Point Cost Estimates provided for the all risks Full risk exposure (all risks happen at full value) million 27

28 Example Based on Real Data Cost Analysis - Details Programme Management Design Development Test & Integration Purchases Production Upgrade to 47 Aircraft Initial Support 28

29 Example Based on Real Data Cost Analysis Top-down ( Million) The Top-down cost evaluation will arbitrarily use an 80% Confidence Level Schedule months 14.4% increase over 70 month programme Top-down Cost Analysis Lets Call It Option 1 Factor All Costs All Costs = x 14.4% = Option 2 Factor In-House Costs In-house Costs = x 14.4% = Purchased Items = = Total = Option 3 Factor Development Integration & Test Costs Development = x 14.4% = Other In-house = = Purchased Items = = Total =

30 Example Based on Real Data Cost Analysis Bottom-up Baseline Cost (excluding risk and other contingencies) million 3-Point Cost Estimates provided for the all Baseline cost drivers Absolute minimum million Most Likely million Absolute maximum million 3-Point Cost Estimates provided for the all risks Absolute minimum million (No risks occur!) Expected Value million (Finance Like it!) Absolute maximum million (Big Number!) Monte Carlo Analysis Uncertainty Analysis (sensitivity around the Baseline Estimate) Risk and Opportunity Analysis (against the Risk Register) Risk Opportunity and Uncertainty Analysis (all data) 30

31 Example Based on Real Data Cost Analysis Monte Carlo Output Uncertainty 31

32 Example Based on Real Data Cost Analysis Monte Carlo Output Risk (and Opportunities) 32

33 Example Based on Real Data Cost Analysis Monte Carlo Output Risk Opportunity and Uncertainty 33

34 Example Based on Real Data Cost Analysis Summary Outputs (Interim View) Confidence Level Million Confidence Optimistic Range % Pessimistic Range % Recommendation % Bottom-up Functional Request Risk Modelling 50% Confidence level Balancing Number for Total Cost Build Up Million % ERF Returns % Technical Risk Contingency % Management Contingency % Total % 34

35 Example Based on Real Data Cost Analysis Monte Carlo Outputs Confidence Level Uncertainty K Risk K ROU K 0% 26,358-26,393 1% 27,062-27,273 5% 27,535-27,771 10% 27,812-28,064 15% 27,997-28,256 20% 28,140-28,404 25% 28, ,549 30% 28, ,677 35% 28, ,786 40% 28, ,898 45% 28, ,018 50% 28, ,130 55% 28, ,243 60% 29, ,359 65% 29, ,483 70% 29, ,605 75% 29, ,736 80% 29, ,890 85% 29, ,071 90% 29, ,315 95% 30, ,642 99% 30,710 1,223 31, % 31,788 1,783 32,377 To Excel Slipping & Sliding Template 35

36 Example Based on Real Data 36

37 Example Based on Real Data Cost Analysis Summary Outputs Bottom-up Approach Top-down From Excel Slipping & Sliding Template Uncertainty Risk/Opp ROU Stretched Equivalent % 50% 79% 50% 37

38 Example Based on Real Data Cost Analysis Summary Outputs (Revised View) Confidence Level Million Confidence Optimistic Range % Pessimistic Range % Recommendation % Bottom-up Functional Request Risk Modelling 50% Confidence level Balancing Number for Total Cost Build Up Million % ERF Returns % Technical Risk Contingency % Management Contingency % Total % 38

39 Example Based on Real Data Integrated Cost and Schedule Using all the previous datasets we can actually do a proper Integrated Monte Carlo Cost and Schedule Analysis We have the technology! 39

40 Cost Including Risk ( K) Example Based on Real Data Integrated Cost and Schedule 31,000 30,000 29,000 28,000 27,000 26,000 25,000 24, Duration (Months) 40

41 Example Based on Real Data Potential Cost Outcome ( Million) Estimates We could Use Top-down Option 1 Crude factor applied to the whole costs = Option 2 Crude factor applied to in-house costs = Option 3 Crude factor applied to Development activities = Bottom-up Risk Opportunity and 80% confidence Level = Slipping and Sliding Risk Opportunity and 50% confidence Level = Finance View Most Likely estimate plus Expected value (Most Likely) = Most Likely estimate plus Expected value (Max) = Integrated Cost and Schedule Pick a dot in middle of the scatter plot! =

42 Example Based on Real Data Actual Cost Outcome So what really happened then! Headline Cost out-turn using agreed pricing rates = However we did not do what the original programme said! There were problems with the RAF delivering aircraft to the factory not a risk identified excluded from the proposal as GFE Return To Works (RTW) programme changed to a Contractor Working Party (CWP) CWP do not attract the same charging rate as RTW CWP use RAF facilities Free of Charge and is therefore cheaper Actual costs based on CWP undertaking work =

43 Example Based on Real Data Actual Cost Outcome Cost Centre Baseline Estimate M Outturn Cost M Programme Management Baseline Cost Design Development Test & Integration Purchases Production Upgrade to 47 A/C Initial Support Grand Total

44 Example Based on Real Data Baseline Estimate Outturn Cost M Programme Management Design Development Test & Integration Purchases Production Upgrade to 47 A/C Initial Support 44

45 Cost Including Risk ( K) Example Based on Real Data Integrated Cost and Schedule 31,000 30,000 29,000 28,000 27,000 26,000 25,000 24, Duration (Months) 45

46 Slipping & Sliding - A Pragmatic Aid to Judgement Conclusion Cost and Schedule Integration the Right Thing to do High degree of correlation between them Heat Maps look good but do they really help? Still suffer from optimism bias Do not allow for the Rumsfeld Factor! Slipping and Sliding Technique is a quick and easy alternative to bridge the gap between optimism and pessimism bias Makes some allowance for the Rumsfeld Factor Easier to interpret than a Heat Map It is not a replacement for Judgement it s an aid Opportunities for improvement Cross-match several points? Slip and Slid between 20:80 or 50:80 Confidence levels (instead of 0:80 or 0:75) 46