BASIC COST RISK ANALYSIS: USING CRYSTAL BALL ON GOVERNMENT LIFE CYCLE COST ESTIMATES

Transcription

1 Proceedings of the 2007 Crystal Ball User Conference BASIC COST RISK ANALYSIS: USING CRYSTAL BALL ON GOVERNMENT LIFE CYCLE COST ESTIMATES ABSTRACT R. Kim Clark Booz Allen Hamilton 700 N. Saint Mary s St., Suite 700 San Antonio, TX USA Requirements today necessitate the need to include cost risk analysis in all government life cycle cost estimates. In particular, the Air Force Cost Analysis Agency has developed a Cost Risk Analysis Handbook to be used as a guideline when conducting basic cost risk analysis. Crystal Ball software is an excellent choice as a tool to be used for cost risk analysis on government cost estimates, particularly when cost models are developed in Microsoft Excel. Crystal Ball allows for the accomplishment of basic cost risk analysis on life cycle cost estimates. Those basics include selecting the distribution type for an identified risk, determining the input parameters to fit the distribution selected, completing the correlation matrix, running the simulation, allocating the risk dollars back to the appropriate line items, and running final reports on the analysis. 1 BASIC COST RISK ANALYSIS Cost risk analysis has emerged as a hot topic the last few years in government cost estimating circles. It has always been considered a way to better determine total program costs in order to reserve necessary funds for those issues or risks that will inevitably appear in every program that exists. However, the question always concerns the probability of that risk occurring and, if it occurs, how severe will be the impact. What will be the overall cost to the program? Cost estimators and program managers recognize the fact that risks will occur in every program. Can estimators develop a standard process for risk analysis that is sufficient enough for an average practitioner and that identifies the amount of risk dollars needed in a program? Yes, they can. Crystal Ball is one tool that has given them the capability to create a cost risk analysis process. The attempt with this paper is not to determine a required method for cost risk analysis but to only recognize that, through one cost risk analysis approach, Crystal Ball is a simulation tool that can assist in fulfilling a specific need - performing cost risk analysis on government life cycle cost estimates. 1.1 History of Cost Risk Analysis in Estimating Risk analysis has eluded the government cost estimating communities for years in its attempt to develop a useable, reliable cost risk analysis method (or process) that can also be easily understood and utilized by the typical practitioner of government life cycle cost estimating. Practitioners of statistical analysis have always been capable of calculating those necessary risk formulas for running a Monte Carlo simulation, etc but the typical estimator has never grasped that level of statistical capability in order to accomplish the effort. Cost risk analysis, in the previous years (the last 20 or so) has often been addressed, but from a different perspective. Risk ranges was the risk analysis term that was most often used when providing a final life cycle cost estimate calculation for the government. A point estimate was provided with a risk range identified based on the feel of the program manager or cost estimator or based on some guidelines as to the degree of detail utilized in developing the cost estimate. At UCLA, there is a class taught called Cost Estimating and Economic Evaluation of Projects (engineering , UCLA extension course). This class provides a chart of the once traditional method of determining risk ranges for project cost estimates. Table 1 shows the three different ranges typically used by a cost estimator for providing an answer to the government on the cost of the project (point estimate) and the possible range the project cost could actually reside in between some lower value and some higher value from the point estimate. For example, if a life cycle cost ROM point estimate was $50 million, then the associated risk range means that the program could cost from $35 million to $75 million. This is not a very promising or assuring prospect for a government program manager knowing that his program cost could move up or down across a range of $40 million.

2 Table 1: Traditional Cost Estimating Risk Ranges Estimate Type Definition Risk Range ROM Rough Order of Magnitude -30% to +50% Without any detailed engineering data. Budget Requires design basis, quotes on high -15% to +30% end items. Definitive Very defined engineering data, quotes, and specs. -5% to +15% 1.2 The Objective of Cost Risk Analysis Although risk ranges are still used on occasion, particularly if the time frame is limited for providing a cost estimate, the primary methodology for government cost estimating, in attempting to identify program risk dollars, is for the practitioner to use a cost risk analysis process (a simulation) as part of the overall project or life cycle cost estimate. The objective is to determine a risk adjusted cost life cycle cost estimate for a program manager or, at the very least, provide a set of options in the form of a cumulative distribution function that gives the program manager the opportunity to select the level of confidence that he has in the dollar value needed for their programs cost risk requirements. Cost risk analysis typically incorporates cost, schedule, and technical risk as part of the overall risk score provided for a WBS (work breakdown structure) line item when using cost risk scoring to determine the spread of the distribution type selected. However, some practitioners go even farther and attempt to break out the differences between cost risk, schedule risk, and technical risk. In either case, the primary goal is a final risk-adjusted cost estimate that a program manager is confident in providing as their government life cycle cost estimate. The search for the specific methodology that all can practice continues today. 2 CURRENT PRACTICE Most individuals involved in government cost estimating are looking for a reasonable process to conduct cost risk analysis as part of their overall life cycle cost estimating practice. That process has to include a tool that provides them with the capability to conduct cost risk analysis. There are tools available for specifically addressing cost estimating needs, but very few, of those cost estimating tools actually incorporate a cost risk analysis function or capability within the model. Most cost risk analysis is performed with a separate tool built specifically for accomplishing Monte Carlo or Latin Hypercube simulation, the major statistical function necessary for cost risk analysis. Crystal Ball is one tool capable of fulfilling the government s need for accomplishing the cost risk analysis function for life cycle cost estimating. The more current effort in government cost estimating practice has been a scramble for practitioners to find a way to accomplish cost risk analysis. There have been several papers presented at SCEA (Society of Cost Estimating and Analysis) and ISPA (International Society of Parametric Analysts) national and local conferences and articles written in society journals. However, the search has steadily continued with no one clear cost risk analysis methodology coming to the forefront. There is a continued attempt today at finding a useable method that can be shared by all practitioners in the field. There have been methods developed to run cost uncertainty then cost/technical/schedule risk analysis using two separate Monte Carlo simulations, methods developed to run three simulations separately for cost, then schedule, then technical risk, methods that declared only one simulation should be run for any risk involved in the program, and of course, methods of running that one simulation different ways. No clear method has stood out, and no real method proven emphatically incorrect in its process and/or capabilities. All methods have followed statistical methodologies to get to an estimated, statistically analyzed, amount of acceptable cost risk for a program. 2.1 Air Force Cost Risk Analysis Handbook Just recently, the Air Force Cost Analysis Agency (AFCAA) produced a new handbook titled Cost Risk Analysis Handbook. This is the Air Force s (AF) attempt to provide direction in utilizing a method for practicing cost risk analysis on government life cycle cost estimates. It provides reasonable methodologies to practice cost risk analysis, and the rationale for selecting certain criteria for modeling risk analysis. Even the AF provides an option to pick between using different methods to con-

3 duct cost risk analysis. The AF handbook gives three alternatives to conducting cost risk analysis: the inputs method, the outputs method, and the scenario method (a non-simulation method). At the very least, the cost analysis agency is attempting to winnow the discussions to a point much more defined to available risk analysis options so that the cost estimating community may actually come to agreement upon a methodology approach to cost risk analysis (or at least some of the community). The inputs simulation method is an approach to cost risk analysis that utilizes the inputs to the calculations contained within the WBS items, which gives the analyst more flexibility in selecting specific data to include as part of risk. The outputs simulation method uses the resulting final calculations provided for the WBS line items and limits the analyst to select that specific calculated result, on a WBS line item, to use as part of the cost risk analysis. Either method is an acceptable cost estimating practice for cost risk analysis and can be conducted using Crystal Ball as a simulation tool. 2.2 Using Crystal Ball in Today s Estimating Environment Providing cost estimates for government programs rarely is completed today without including cost risk analysis. Risk reserve will usually be included in most government life cycle cost estimates. The core of cost risk analysis still lies in the capability to conduct statistical analysis as random sampling of data within a prescribed distribution shape, i.e. a simulation. Crystal Ball is one tool that can provide the necessary capabilities to conduct a statistical simulation (Monte Carlo or Latin Hypercube) to run any of the methodologies the Air Force Cost Analysis Agency is prescribing to and for other practitioners analysis processes in need of a simulation tool for conducting cost risk analysis. There are other available simulation tools in the marketplace that could likely perform the same functions necessary to conduct cost risk analysis on government life cycle cost estimates or ACE). Crystal Ball is one of the existing tools that not only performs the functions but is also user friendly in its capability for the estimator to interact and develop the necessary inputs for conducting a cost risk analysis. Crystal Ball obviously has other functions within it s software that will not be discussed in this paper and are left for the reader to investigate. It is not the author s intention to induce your consideration of one tool over another but just to illuminate the capability of the Crystal Ball tool in this circumstance as an optional tool to perform statistical analysis. It is the primary functions behind Crystal Ball s simulation capability that is the main piece of the tool used for cost risk analysis on government life cycle cost estimates. Crystal Ball can be a tool of choice for the cost estimating community to use in their everyday necessities of conducting cost risk analysis for government cost estimates. 3 GOVERNMENT LIFE CYCLE COST ESTIMATING PROCESS The practice of government cost estimating has a particular style that has become specific to anyone developing a life cycle cost estimate (or project estimate) for a Department of Defense (DoD) program. The main objective is typically to follow the DoD acquisition framework that brings together costs in phases: concept refinement, technology development, system development & demonstration, production & deployment, and operations & support. The cost estimator is tasked with walking through the estimating process to reach the inevitable goal of a final life cycle cost estimate, including cost risk analysis. 3.1 Develop the Estimate The process of developing life cycle cost estimates, or an estimating process in general, has likely been drawn and re-drawn thousands of times. Figure 1 shows the example of a basic simplified approach to starting a life cycle cost estimate (LCCE) and eventually getting to a final resulting LCCE that can be presented to a program manager for budget submittal. The basic process for developing a government LCCE will usually consist of developing a program WBS, then gathering the data to provide the information needed for that WBS breakout, and then finally accumulating to a first initial LCCE. Of course it isn t quite done yet. You get a few more changes that you believe are it, so you go ahead and run your cost risk analysis to get a new risk adjusted LCCE. However, more changes just came in, so you then re-do your LCCE, re-run your cost risk analysis, and the task continues in that mode until such time you reach your deadline and changes quit coming. The final result of the process is a risk adjusted LCCE that has been inflated to a program cost useable as a new budget. However, this paper is not about the LCCE process but about the single square in the process called cost risk analysis.

4 Refine/Update LCCE Develop Program WBS Gather Data Initial LCCE (BY$) Risk Adjusted LCCE (BY$) Final LCCE (TY$) Conduct Cost Risk Analysis Figure 1: Life Cycle Cost Estimating Process 3.2 Run Cost Risk Analysis A simplified approach to the cost risk analysis process is shown in Figure 2 where there are five basic steps identified to accomplishing a cost risk analysis process using Crystal Ball as the primary simulation tool when performing government life cycle cost estimates. These are by no means the only steps or only process that may be identified as a cost risk analysis activity but just a high level notional concept of basic steps to accomplish the process. The objective for this section is to discuss the primary topic of the paper: a basic approach to cost risk analysis using Crystal Ball in government life cycle cost estimates. There are five basic steps identified as a minimum requirement to conducting cost risk analysis using Crystal Ball. Those steps are: determine risk ratings, set up Crystal Ball, develop the correlation matrix, run the simulation, and select the confidence level results. These steps are best followed sequentially, but some of them can be accomplished out of sequence. Determine Risk Ratings Setup Crystal Ball Defaults Crystal Ball WBS Assumptions Forecasts Run Simulation Select Confidence Level Develop Correlation Matrix Figure 2: Cost Risk Analysis Process using Crystal Ball Determine Risk Ratings Once the WBS is structured as necessary for the LCCE, some of those line items will need to have risk identified as part of the overall cost. The line items identified and the associated risk ratings or scores are usually provided by the engineering team working on the project. The risk is scored by evaluating both probability of risk and impact of risk for a final single rating typically given as low, medium, or high (and sometimes very high). The cost estimator will usually develop a risk matrix to assist the engineers in scoring the separate line items for risk. Figure 3 shows an example of a risk matrix used for risk scoring. The different scores and colors represent the different risk ratings of L, M, and H.

5 PROBABILITY RISK SCORING MATRIX Near Certainty Highly Likely Likely Unlikely None None Low Moderate High Very High IMPACT Figure 3: Risk Scoring Matrix Setup Crystal Ball Once the risk scores have been determined, the next step is to begin to set up the Crystal Ball software for a simulation run. The first necessary activity is to determine the assumptions within the WBS based on the outputs method of cost risk analysis. That entails selecting the WBS line items identified with risk scores and determining which type of distribution will be selected for that specific line item. The most common distribution types selected are the log normal curve, the triangular curve, the normal curve, and the uniform curve. However, Crystal Ball gives you the option to select from 21 different distribution types or to select only the six basic distribution types. The basic distribution types are shown in Figure 4 below. Figure 4: Crystal Ball Basic Distribution Types The selection of the distribution type is necessary because the additional parameters for running the simulation must be calculated or gathered before Crystal Ball can be set up. For example, for a triangular curve, you need three parameters: the most likely value, the minimum value, and the maximum value. For a lognormal or normal curve, you need the mean and standard deviation values. Some of these values can be provided by the engineers on the project, but most will have to be calculated based on default calculations. Figure 5 shows an example of selecting default values for calculating minimum, maximum, and standard deviation values based on the risk scores assigned to a line item and the type of distribution selected. The values for the lognormal and normal curve are for a percentage of the most likely value to determine the standard deviation. The percentages for the triangular curve are factors for the minimum and maximum amounts based on multiplying that change times the most likely value. For example, the medium triangular curve is a -10% and +50% for the minimum and maximum values. The minimum amount is 90% of the most likely value, and the maximum amount is 150% of the most likely value. Once these calculations are complete, based on the distribution type and risk score, the calculated values must be turned to values to eliminate the formula in order to prepare for the simulation run. Next, place the curser on the cell with the WBS output value, identified as the most likely value, and then select the assumption icon on the Crystal Ball tool bar to identify that cell value as an assumption for the cost risk analysis. Crystal Ball will ask you to select the distribution type, which you now know, and then identify the appropriate parameter values. Also be sure to select the name of the line you have selected and do not leave the cell value that Crystal Ball has defaulted to. Each of

6 these selections is made by putting an equal sign in the input screen and selecting the appropriate cell in Excel where the value is residing. The cell will now turn green, showing that you have identified that cell as a Crystal Ball assumption. Risk Distribution Defaults LogNormal Curve Low 25% Med 35% High 45% Triangular Curve min max Low -10% 10% Med -10% 50% High -10% 100% Very High -10% 200% Figure 5: Risk Distribution Type Default Factors Since you used the equal sign in your identification of values when selecting the cells those values reside in, you can now copy and past the assumption setup by using the Crystal Ball copy and past functions on their menu bar. It is critical to realize these are not the Excel copy and past functions. You can copy a triangular distribution to another triangular distribution line item but not across different distribution types. For example, a lognormal curve can be copied to a lognormal identified line item but not to a normal curve item, etc. Once all assumptions are selected and their Excel cells turned green, it is time to select the forecasts. The forecasts are the line items where you would like to see the simulation results. It is usually beneficial to identify those line items being selected for forecasts to be placed next to the line items identified as assumptions. The reason for this is that some line items will be both assumptions and forecasts. When you select the assumptions and then select the same item as a forecast, the forecasts turn blue and is now covering up the fact that that line item was a assumption previously colored green. Crystal Ball knows it is both but if you can t see both colors, you may not remember that that line item is both an assumption (green) and a forecast (blue). For government cost estimating, at a minimum, the second level WBS line items are chosen for obtaining the forecasted results selected as forecasts for the simulation run. The minimum number of forecasts to be selected has to be one so that you can get a total program cost as an output of the Crystal Ball simulation run. However, the most forecasts selected can be each line item that exists within the WBS setup. Crystal Ball suggests a maximum amount of 500 assumptions on a single worksheet and about the same number of forecasts in the new v7.x software. The objective of selecting forecasts is to create the visibility necessary to get to the level of detail you would like to report for cost risk analysis purposes. Once you establish your first forecast item (using the equal sign again to identify the name) you can use the copy and past function from Crystal Ball and copy to all other items selected to be forecasts. Now both assumptions, the green cells, and the forecasts, the blue cells, have been identified and prepared for a Crystal Ball simulation run. Figure 6 shows an example of the selection and colors of assumptions and forecasts next to the project WBS with the forecasts moved off to the right of the assumptions.

7 PROGRAM CB LCCE - COST RISK ANALYSIS 3/1/07 Risk Analysis Setup (BY07$K) WBS Description Total Assumptions Forecasts TOTAL PROGRAM COSTS 371, , , Concept Refinement 1, , , Concept Studies/Research CR SE/PM Technology Development 13, , , Technology Analysis 8, , TD SE/PM 4, , System Development & Demonstration 112, , , CB System Development 63, , , Software Development 53, , Aerospace SW Development 34, , Ground SW Development 15, , Ground Mgmt SW Dev 3, , Hardware Development 7, , Aerospace Units 2, , Aerospace EDM 2, , IA&T Ground Units 4, , Ground EDM 3, , IA&T ST&E 2, , Operational Site Activation 1, , , Test Equipment Development 6, , , Training Development 4, , , Data 3, , , SD&D ECPs SD&D SE/PM 33, , , Develop Correlation Matrix Figure 6: Assumptions and Forecasts Correlations are a critical part of running a cost risk analysis and are often forgotten as part of the setup. Figure 7 shows the difference in simulation results when correlation is not included. As part of the setup for the assumption selections, one of the options when selecting the assumption is to provide correlation values between line items on the WBS. The assumption selection process allows you to enter those values manually. However, there is a much easier way to deal with correlations and that is to run the Correlation Matrix tool in Crystal Ball. Probability No Correlation 0.00 Cost Dollars Million With Positive Correlation Figure 7: Example of Correlation Effects on Simulation Results Before the correlation matrix is built, the cost analyst, along with the engineers on the project, will need to determine the different correlations between the WBS line items. Typical correlations are hardware to hardware and software to software, etc... If you have multiple hardware procurement or software development lines, than those lines would be correlated at some relational factor that ranges from -1 to +1. Notional examples are to use.7 as a software correlation,.5 as a hardware correlation, and then.3 as a default correlation. Correlations tell you that for every change on one line item, that change will have

8 an affect on another line item. The amount of correlation is the amount of affect incurred. A default is typically included in the correlation matrix since when any activity on the program is changed, it will usually have an affect on most of the other line items or activities as well. The AF Cost Risk Analysis Handbook recommends.5 as a default correlation and provides other examples of what correlation levels should be used. To set up the correlations, select the Correlation Tool within Crystal Ball and walk through the directions by selecting some of the WBS line item assumptions or all of the WBS line item assumptions. Crystal Ball builds the correlation framework for you but you have to then enter the correlation amounts. Once you have all the cells filled on the correlation matrix, click the Load the matrix button, and the correlations are automatically loaded into the assumptions. After loading the correlations, you can always select a specific assumption and manipulate the correlations by hand if you would like to change one. Figure 8 shows a notional example of a partial correlation matrix in Crystal Ball. Load the matrix Concept Studies/Research (Risk Analysis) CR SE/PM (Risk Analysis) Technology Analysis (Risk Analysis) TD SE/PM (Risk Analysis) Aerospace SW Development (Risk Analysis) Ground SW Development (Risk Analysis) Concept Studies/Research (Risk Analysis) CR SE/PM (Risk Analysis) Technology Analysis (Risk Analysis) TD SE/PM (Risk Analysis) Aerospace SW Development (Risk Analysis) Ground SW Development (Risk Analysis) Ground Remote Mgmt SW Dev (Risk Analysis) Aerospace EDM (Risk Analysis) IA&T (Risk Analysis) Ground EDM (Risk Analysis) IA&T (Risk Analysis) ST&E (Risk Analysis) Operational Site Activation (Risk Analysis) Test Equipment Development (Risk Analysis) Training Development (Risk Analysis) Data (Risk Analysis) SD&D ECPs (Risk Analysis) SD&D SE/PM (Risk Analysis) Ground Remote Mgmt SW Dev (Risk Analysis) Aerospace EDM (Risk Analysis) IA&T (Risk Analysis) Ground EDM (Risk Analysis) IA&T (Risk Analysis) ST&E (Risk Analysis) Operational Site Activation (Risk Analysis) Test Equipment Development (Risk Analysis) Training Development (Risk Analysis) Data (Risk Analysis) SD&D ECPs (Risk Analysis) SD&D SE/PM (Risk Analysis) Run Simulation Figure 8: Correlation Matrix At this point, you are pretty much ready to run your Monte Carlo simulation in Crystal Ball. However, before you jump in to the simulation, you should always check your Crystal Ball defaults and determine if Crystal Ball is set up the way you want it to be for your simulation. Select the Run Preferences from the Crystal Ball tool bar and look at the default options Crystal Ball gives you. Particularly view the number of trials, since Crystal Ball defaults to 1,000 trials you would want to change that value to 10,000. The rest of the defaults will need to be reviewed and determined based on your specific simulation needs. Now select the Run button and begin the simulation. It is helpful to select, as an option, the chance to view the simulation as it is running by looking at the distribution curves being created. If you look at the total program distribution curve, you can tell right away if it looks like your simulation is running well. Check the center value on the distribution curve and that approximate value should be something you are expecting as a value that is close to your total most likely program cost. If it isn t, then you can stop the run instead of letting it continue and wasting your time. If it doesn t appear correct, you will need to start troubleshooting your setup and see where you have made a mistake. One of the most common mistakes is in the assumptions. The assumptions can t be formulas, they must be a value. But also, the line items not selected as assumptions must then be formulas that add up those values to get to a final program cost Select Confidence Level Result Finally our Crystal Ball has shown us the light. Once the simulation is completed, you now actually have a series of results. For government life cycle cost estimating, one of the options we would have set up was to view the results of the simulation run in 10% intervals. Figure 9 and Figure 10

9 are views of a forecast that shows the distribution results and the actual program costs based on the 10% intervals of the confidence levels. Historical data has shown that previous government programs have estimated costs around the 30% confidence level, which means that the cost will probably be that much or less at that confidence level. Figure 10: Resulting Cumulative Distribution Function Figure 9: Resulting Lognormal Frequency Distributions Total Program Percentiles Costs 0% 260, % 337, % 356, % 371, % 384, % 397, % 411, % 427, % 446, % 476, % 680,981.7 Figure 10: Resulting Cumulative Distribution Function and Percentiles Most government programs select the 50% confidence level for their total program costs in life cycle cost estimating and some even select the 80% confidence level. The AF cost risk analysis handbook recommends 60% since that value reflects total costs that are closer to the average total cost calculated for that simulation. This is due to the inevitability of the program costs resulting in a lognormal curve as the final results. A lognormal curve, as shown in Figure 11, for government cost estimates are typically skewed to the right. Therefore the mean, median, and mode work in reverse direction. The mean is the value farthest to the right on a right skewed log normal curve. The 50% value, chosen by most government programs, is actually just the middle value of the results and is less than the average amount, on a lognormal curve. At that confidence level you are saying that you have a 50% chance that the total program costs will be that much or less but you are also saying that the total costs selected are less than the average costs resulting from this simulation. The mode is, of course, the value that was repeated the most and is the left most value of the three.

10 Figure 11: Lognormal Curve s Mean, Median, and Mode 3.3 Present Results Once you have identified your confidence level, the final step is to allocate those costs back down throughout the complete WBS structure in order to finish with a complete set of program costs, by line item that now includes risk. At this point, you can present the results and will need to show both the values as selected, at the 50% confidence level (if that is the selection) and the actual spread of results in case the program manager would like to see their options or sensitivity to the actual risk results. A good example is to show the results in what we usually refer to as an S curve. It is just a duplication of the cumulative frequency distribution provided by the Crystal Ball results but is typically re-drawn in Excel for a clearer picture for the program manager. Figure 12 shows an example of a notional S curve charted in Excel. Percentile 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% PROJECT S CURVE $20,483 $22,599 $24,291 $21,690 PE (31%) $17,000 $19,000 $21,000 $23,000 $25,000 $27,000 $29,000 Value (BY04 $M) $27,946 Figure 12: Notional Typical Project S Curve In addition, at this time, it is crucial that the simulation run be saved and all resulting data also be saved for future reference. If the results of the simulation run are saved, you can re-build the simulation any time it is necessary to re-look at the resulting simulation data. For government cost estimating, you would specifically save the simulation run, the simulation, report, the percentiles by forecasts, the statistics by forecasts, and the correlation matrix to be certain that this simulation can be repeated and be auditable. It is critical that government estimates calculated using acceptable cost estimating methods, that it be fair and reasonable, have an audit trail for defending the data, and that the estimate can be repeatable. Having conducted the process and saved the data, then completes the cost risk analysis activity. You now have a risk adjusted cost estimate to continue with as a gov-

11 ernment life cycle cost estimate. From this point you would inflate the risk adjusted cost estimate from base year dollars to then year dollars in order to have a program costs for budget submittal. 4 CONCLUSION The adoption of Crystal Ball software for conducting cost risk analysis on government life cycle cost estimates has been of extreme benefit due to the ease, user friendliness and powerful capability that the COTS tool gives the cost estimator. What was previously considered to be a horrendous undertaking, performing cost risk analysis to the point that a simple risk range methodology was employed, has now become standard practice due to the availability of software tools to accomplish this necessary task. In order to conduct basic cost risk analysis for government cost estimates, Crystal Ball sufficiently covers the requirements and can continue to provide the support necessary to achieve cost risk analysis results that any analyst may be required to perform. 5 REFERENCES Anderson, T. P. (2004). Cost risk tutorial. Space Systems Engineering and Risk Management Symposium. Aerospace Corporation, El Segundo, CA. Bolles, M. (2003). Understanding risk management in the DoD. Acquisition Review Quarterly (Spring), Book, S. A (2000). Estimating probable system cost. Crosslink.(Winter), Clark, K., & Clish, J. (2005). Cost risk analysis process incorporating risk management office risk scores Joint ISPA/SCEA International Conference. Booz Allen Hamilton, Colorado Springs, CO. Cost Risk Analysis Handbook (2006). Force Analysis Division, Air Force Cost Analysis Agency, USAF. Dechoretz, J., Book, S., & Hofmann, J. (2004). Cost risk analysis: completing the estimate. Training and Case Study for ASC/FMCE. MCR Federal, Inc. Pritchard, C. L. (2001). Risk Management Concepts and Guidance. 2 nd edition. ESI International, Arlington, VA. Remer, D. (2003). Cost estimation & economic evaluation of projects. Department of Engineering, Information Systems, and Technical Management. UCLA Extension Course, Engineering BIOGRAPHY R. Kim Clark (clark_kim@bah.com, ) is an associate with Booz Allen Hamilton in San Antonio and works in the areas of life cycle cost estimating, business case analysis, cost risk analysis, and economic analysis. His eighteen years of cost experience have been on aircraft and space programs for the Air Force, including joint programs, and NASA. He has associate degrees in Avionic Systems, Math, and Electronic Engineering, a BS in Industrial Technology, and a MBA in Operations Management. He is a certified cost estimator/analyst (CCE/A) through SCEA and a certified parametric practitioner (CPP) through ISPA.