Understanding the Results of an Integrated Cost/Schedule Risk Analysis James Johnson, NASA HQ Darren Elliott, Tecolote Research Inc. 1
Abstract The recent rise of integrated risk analyses methods has created a new opportunity for complex projects to understand the dynamic interrelationship of cost, schedule, and risk. NASA has been implementing Joint Confidence Level (JCL) analysis of cost, schedule, and risk for all major projects since 2009 in an attempt to proceduralize and codify the requirements for an integrated risk analysis product. Since the method of integrated risk analysis is itself new, the visualization of analysis results has been challenging. Starting with the simple X-Y scatter plot, this paper will explain how to understand the results of an integrated cost/schedule risk analysis using illustrative examples. The authors will also propose new advanced methods for visualizing results, and provide evidence of the valuable insight that can be gained from conducting an integrated cost/schedule risk analysis. 2
NASA s Probabilistic Budgeting Policy At KDP-B Projects must generate a low and high cost and schedule estimates with associated probabilities of completing at or below those costs/dates An independent SRB will evaluate project-generated results Decision authority will decide upon the low and high cost and schedule targets. Goal is to set budgets at a higher probability of success in order to give projects a better chance of success at KDP-C At KDP-C Projects must generate a cost-loaded schedule and produce a JCL I from an Integrated Risk Analysis that is executable within the available annual resources An independent SRB will evaluate the project-generated JCL results and model Decision Authority will decide the JCL (probability) for the associated development and life cycle cost at which the agency commits to deliver the project *see backup for some clarifications 3
Evolution of Confidence Level Analysis at NASA Project Estimates (Advocacy) Project Primarily Bottoms up Point Estimates and Cost Confidence Levels Joint Cost and Schedule Confidence Level (JCL) 2002 and before 2003-2007 2/2007 1/2009 5/2009 12/2009 11/2010 8/2012 Assessment Parametric estimates and Assessment Formalized Cost Confidence Level Policy (2006) Parametric-Based Confidence Levels JCL Policy Established Cost-Loaded Schedule Requirement Established Refined Requirements for KDP-C Established Assessment of Project JCLs Add KDP-B Confidence Levels for cost & schedule ranges NPR 7120.5E Effective KDP-B cost & schedule probabilistic ranges Independent Estimates (Non-Advocacy) 4
What is an Integrated Risk Analysis? TI $ TI = Time-Independent Cost, e.g., Materials TI $ TD $ = Segment Duration X Burn Rate Project Start TI $ TI $ Uncertainty Probability of Occurrence Duration Uncertainty TI $ Project End Task Duration Risk Register TI $ TI $ TD $ = Segment Duration X Burn Rate TD = Time-Dependent Cost, e.g. marching army cost Burn Rate Uncertainty 5
Traditional S-Curve Reports S-Curves provide ranges and overall statistics for a metric S-Curves provide no insight into how the results relate to each other Insufficient for analyzing an integrated risk analysis S-Curves do not show relationship between cost and schedule 6
Integrated Risk Result Scatter Plot Vertical bar indicates x-axis (Schedule) Confidence level Points represent cost/schedule result pair from each iteration of the simulation Frontier line identifies cost/schedule surface that meets Target JCL Lower quadrant identifies Iterations satisfying cost and schedule targets ~ the JCL Horizontal bar indicates Y-axis (Cost) confidence level Scatter Plot displays the overall cost/schedule range and the relationship 7 between cost and schedule results 7
Integrated Risk Result - Milestone Overlay Blue Line shows Cumulative funding authority for the project Green line shows Cumulative cost plan Scatters show the cost and schedule range for project identified events Triangles display start Date of discrete risks In the plan Overlay displays cost and schedule results over time for selected key program events and/or milestones, as well as risks and funding level 8
Integrated Risk Result Schedule Criticality Shows how frequent a task is on the critical path of the schedule, helps support identify tasks that may not be on primary Critical Path but are drivers 9
Integrated Risk Result Discrete Risk Criticality Shows how frequent a discrete risk task occurs on the critical path when the risk is active, helps support identification of risks that could be drivers 10
Integrated Risk Result Duration and Cost Sensitivity Shows how correlated cost or duration are to a target metric (e.g., total duration, total cost), this identifies items that are potential drivers 11
Integrated Risk Result Tail Contingency Displays the delta from the point estimate to statistical values (mean, 70%, 95%), This identifies items which have overrun potential (Cost or duration) 12
Integrated Risk Result - S Curve Comparison Assessing the impact of uncertainty and risk assumptions to cost and or duration results is an important analysis S-Curve comparison charts support this analysis very well Shows how the S-Curve for a metric (cost or duration) is affected by assumptions within a scenario, this data can be used to gauge impact of inputs 13
Integrated Risk Result Annual Cost Uncertainty Reserve Utilization Shows the annual cost risk results in comparison to available resources and confidence level target. Helps to identify potential shortfalls and reserve needs,14
Path Forward at NASA JCL, the results, and graphics are still very new at NASA Ongoing process of management education Includes both internal, external, and review teams JCL Brochure created and posted on CAD website: http://www.nasa.gov/offices/ooe/cad.html JCL Appendix currently in draft with release in Summer 2013 (along with main CEH) Directorates are codifying and developing standardized process for presentation of results to NASA management councils 15
Conclusion Integrated Cost and Schedule Risk analysis can provide meaningful results Quantify the relationship between cost and schedule Quantify the likelihood of program success for delivering on-time and on-budget Identify the risk drivers affecting program success Identify interim milestone results and timing of discrete risks in comparison to available resources Identify the relative contribution of discrete risks (known s) and uncertainty (unknowns) Evaluate the annual cost phasing, including joint cost schedule risk, in comparison to available resources NASA Leadership is Getting Benefits from Joint Confidence Level 16
Thank You!
NASA Estimating Policy Clarification Resource Loaded Schedule The policy clearly states that the projects are required to generate a resource-loaded schedule NASA s definition of resource loading is the process of recording resource requirements for a schedule task/activity or a group of tasks/activities The usage of resource loading implies, to many people, that the tasks need to be loaded with specific work or material unit resources This is NOT the intent of the policy The intent of the JCL policy is not to recreate the lower level management responsibilities of understanding and managing specific resources (labor, material, and facilities) but to instead model the macro tendencies and characteristics of the project Cost loading a schedule is sufficient and a resource loaded schedule is not required. 18
NASA Estimating Policy Clarification Risk Informed The policy states that a project will need to perform a riskinformed probabilistic analysis to produce a JCL The terminology risk-informed can be ambiguous on interpretation. NPR 7120.5 defines Risk as the potential for performance shortfalls, which may be realized in the future, with respect to achieving explicitly established and stated performance requirements Typically, from a risk management perspective, discrete risks are identified, tracked, and mitigation plans are formulated By risk-informed, the policy is stating that all appropriate discrete risks be modeled, but it is also the intent of the policy for risk-informed to account for various uncertainties (that may not be discretely managed in the risk management system including estimate and model accuracy). 19
NASA Estimating Policy Clarification Life Cycle Cost and Schedule It should be made clear that the scope of the JCL analysis typically includes content only through Phase D This, by definition, is not the total lifecycle costs (as defined by NPR 7120.5E) or operational life of a project The exact content of what is included in the costs and schedule to meet this Phase D requirement needs to be agreed upon between the project and the nonadvocacy entity, as typically defined in the Terms of Reference (ToR) for the Non-Advocate Review 20