Multidimensional RISK For Risk Management Of Aeronautical Research Projects

Size: px

Start display at page:

Download "Multidimensional RISK For Risk Management Of Aeronautical Research Projects"

Darrell Lyons
5 years ago
Views:

1 Multidimensional RISK For Risk Management Of Aeronautical Research Projects RISK INTEGRATED WITH COST, SCHEDULE, TECHNICAL PERFORMANCE, AND ANYTHING ELSE YOU CAN THINK OF Environmentally Responsible Aviation May 0 0

NASA Uses Two Complementary Processes For Risk Management Risk-Informed Decision Making (RIDM) Emphasizes the proper use of risk analysis to make risk-informed decisions that impact all risk

2 NASA Uses Two Complementary Processes For Risk Management Risk-Informed Decision Making (RIDM) Emphasizes the proper use of risk analysis to make risk-informed decisions that impact all risk dimensions including safety, technical, cost, schedule, etc Acknowledges the role that subject matter experts (SMEs) play in decisions. Emphasizes that the cumulative wisdom provided of SMEs is essential for integrating technical and nontechnical factors to produce sound decisions due to the availability of technical data and the complexity of missions Source: NASA/SP NASA Risk-Informed Decision Making Handbook Continuous Risk Management (CRM) To manage those risks associated with the performance levels that drove selection of a particular alternative (from RIDM) A systematic and iterative process that efficiently identifies, analyzes, plans, tracks, controls, and communicates and documents risks associated with implementation of designs, plans, and processes Source: NPR 8000.A Agency Risk Management Procedural Requirements

RIDM Selects Alternatives & CRM Addresses The Implementation Of Alternatives Risk-Informed Decision Making (RIDM) Continuous Risk Management (CRM)

Analysis (Integrated Perspective) and Development of the Technical Basis for Deliberation Risk-Informed Alternative Selection Deliberate and Select an

3 RIDM Selects Alternatives & CRM Addresses The Implementation Of Alternatives Risk-Informed Decision Making (RIDM) Continuous Risk Management (CRM) Identification of Alternatives Identify Decision Alternatives (Recognizing Opportunities) in the Context of Objectives Risk Analysis of Alternatives Risk Analysis (Integrated Perspective) and Development of the Technical Basis for Deliberation Risk-Informed Alternative Selection Deliberate and Select an Alternative and Associated Performance Commitments Informed by (not solely based on) Risk Analysis * Source: NASA/SP NASA Risk-Informed Decision Making Handbook

CRM Uses The Identify Step To Document Risks In The Form of Risk Statements Risk Statements have

Scenario A sequence of credible events that specifies the evolution of a system or process from a

In the context of risk management, scenarios are used to identify the ways in which a system or

4 CRM Uses The Identify Step To Document Risks In The Form of Risk Statements Risk Statements have distinct elements. Scenario A sequence of credible events that specifies the evolution of a system or process from a given state to a future state. In the context of risk management, scenarios are used to identify the ways in which a system or process in its current state can evolve to an undesirable state. Likelihood Probability of occurrence. Consequence The possible negative outcomes of the current conditions that are creating uncertainty Given SCENARIO there is a that, CONSEQUENCE will occur

Cost, Schedule, & Technical Risks Tend To Correlate

increases Schedule decreases may create more technical

Hard technical challenges tend to take longer to

cause schedule slips Example: A facility unavailable

consequence Cost risk due to schedule slip Technical

5 Cost, Schedule, & Technical Risks Tend To Correlate With Each Other Higher costs tend to follow schedule increases Schedule decreases may create more technical risk Tighter schedules tend to have greater cost risk Hard technical challenges tend to take longer to execute and tend to cost more A technical risk may cause schedule slips Example: A facility unavailable for testing represents Schedule risk due to time consequence Cost risk due to schedule slip Technical risk due to impact on a technical, project goal Etc Schedule Technical Cost

6 Risk Is Typically Measured As The Ordered Pair Of (Likelihood, Consequence) = Risk Likelihood Estimation of the likelihood that the risk event will occur Definitions RISKS (Likelihood, Consequence) Definitions Consequence Estimation of the impact to the program if the risk event occurs RISK MATRIX Engineering Judgment Categorization of risks typically follows engineering judgment

Level Managers Use Customized Criteria To Bin Risks Into A Risk Matrix Probability Very Likely Likely Possible Unlikely Highly Unlikely Likelihood Rating Expected to happen Could happen.

Extremely remote possibility that it will happen. Strong controls in place.

7 Level Managers Use Customized Criteria To Bin Risks Into A Risk Matrix Probability Very Likely Likely Possible Unlikely Highly Unlikely Likelihood Rating Expected to happen Could happen. Controls have significant limitations or uncertainties. Could happen. Controls exist, with some limitations or uncertainties. Not expected to happen. Controls have minor limitations or uncertainties. Extremely remote possibility that it will happen. Strong controls in place. RISK MATRIX CONSEQUENCE Very Low Low Moderate High Very High Technical Schedule Negligible or no impact to achievement of Subsonic Transport System Level Metrics, Technical Deliverables, Technology Maturation, or KPP Goals Level Milestone(s): < month impact Level, Milestone(s): month impact Cost Between 0% and % increase over allocated budget (Sub-Project, Element or Task level) Minor impact to achievement of Subsonic Transport System Level Metrics, Technical Deliverables, Technology Maturation, or KPP Goals Level Milestone(s): month impact Level, Milestone(s): month impact Between % and 0% increase over allocated budget (Sub-Project, Element or Task level) Safety Negligible or no impact Could cause the need for only minor first aid treatment Some impact to achievement of Subsonic Transport System Level Metrics, Technical Deliverables, Technology Maturation, or KPP Goals Level Milestone(s): month impact Level Milestone(s): month impact Level, Milestone(s): month impact Between 0% and % increase over allocated budget (Sub- Project, Element or Task level) May cause minor injury or occupational illness or minor property damage Moderate impact to achievement of Subsonic Transport System Level Metrics, Technical Deliverables, Technology Maturation, or KPP Goals Level Milestone(s): > month impact Level Milestone(s): > month impact Level, Milestone(s): > month impact Between % and 0% increase over allocated budget (Sub- Project, Element or Task level) May cause severe injury or occupational illness or major property damage Major impact to achievement of Subsonic Transport System Level Metrics, Technical Deliverables, Technology Maturation, or KPP Goals Level Milestone(s): > month impact Level Milestone(s): month impact Greater than 0% increase over that allocated budget (Sub- Project, Element or Task level) May cause death or permanently disabling injury or destruction of property 6 6

8 The Risk Matrix Provides The Framework For CRM Risk Analysis RISK MATRIX Effective analysis makes it possible to move total project risk from red to green But how do you know you are focused on the right project risks? Focusing on the wrong risks may keep total project risk in the red? 7

9 Current Risk Matrix Development Methods Often Fail To Give A Complete Risk Picture Cost Risk Notional Representation Of Risks In Three Dimensions Schedule Risk Technical Risk Why are we looking at only one dimension at a time? Should we call pt(,,) a Cost Risk, a Schedule Risk, or a Performance Risk? Is pt(,,) more risky than the other points just because it has a high schedule severity? Is pt(,,) just as risky as pt(,,)? What if we have risk across four dimensions? Or five? Or Six? How do we know we are focusing on the right risks? =pt =pt =pt 8

10 MRisk Makes Use Of Anchor Points And Multidimensional-Distance Measure To Determine Total Risk Cost Risk Schedule Risk Technical Risk min (,,) max (,,) The anchor points (,,) and (,,) come from our definition of the consequence scale Distance for each point is defined by the distance of that point from the minimum over the sum of the distance from the minimum and the maximum The distance value explains the precise consequence for each risk regardless of the number of dimensions The greater the distance the greater the consequence and vice versa This procedure is scalable to infinite dimensions of consequence, i.e. (,,, n ) (,,, n ) d = d mind d min + max Consequence Scale L M M M H H H C C C Low Medium High Critical 9

Anchor Points & Mahalanobis Distance Make Risk Analysis Objective & Logically Consistent The anchor points make it possible for us to know relative risk Anchor points allow us to make the distinction

11 Anchor Points & Mahalanobis Distance Make Risk Analysis Objective & Logically Consistent The anchor points make it possible for us to know relative risk Anchor points allow us to make the distinction between a (,,) and a (,,) A cost consequence of, schedule consequence of, and safety consequence of has a distinct distance away from no consequence (,,) and disaster (,,) Mahalanobis Distance keeps decision makers consistent in their thinking. By calculating risk based on the relationship between costs, schedule, safety, etc MRisk identifies when violations of known relationships occur in the risk ranking process For example, cost and schedule have a known relationship in the PM world Schedule Cost Scope 0

MRisk Provides A Complete Risk Picture MRisk addresses several shortcomings in the current methods. MRisk deals with all of the dimensions of Risk simultaneously to provide a complete risk picture.

12 MRisk Provides A Complete Risk Picture MRisk addresses several shortcomings in the current methods. MRisk deals with all of the dimensions of Risk simultaneously to provide a complete risk picture. MRisk makes risk analysis objective and consistent with SME judgment. MRisk provides more advanced statistical algorithms to Risk Management without changing the current processes or products Schedule Scope Cost RISK MATRIX

13 Mahalanobis Distance Mapping Tells Us How Far Each Risk Is From All Of The Other Risks, Thus Highlighting Outliers Notional Data Set From Risk Scoring Point Sched Cost Perf Dim n 0 m Outlier Typical Point Using traditional distance measures the outlier point in the above scenario could be masked by its proximity to the other points Mahalanobis distance highlights the point as an outlier because of its relative distance away from the group Mahalanobis distance accounts for the relationship of each risk to another and highlights the risks that are uncorrelated, thus detecting extreme risks more efficiently d min = (x-x min )Σ - (x-x min ) d max = (x-x max )Σ - (x-x max ) where Σ - is the Inv(Covariance Matrix) x min = [,,] x max = [,,]

program Those observations will have a variance and covariance Any set of random

14 Mahalanobis Distance Is Based On The Interdependencies Of Dimensions Consider two, random variables X and Y that consist of risk observations for some project or program Those observations will have a variance and covariance Any set of random variables will have a Variance-Covariance matrix Obs Obs Obs Obs n Obs Obs Obs Obs n,, =,

15 Two Dimensional Mahalanobis Distance Example Consider again our two, random variables X and Y that consist of risk observations for some project or program with the derived variance-covariance matrix The distance between two points in XY-plane depends on the inverse of the variancecovariance matrix It s simple to expand this case to Schedule Risks(X) vs Cost Risks(Y) vs Technical Risks(Z) or any other type of risk comparison Y 0 =(,) X-Y = (,)-(,)=(0,) Inv(Var-Cov)= = Y X 0 =(,) Distance = (X-Y) (X-Y)` = (0,) (0,)`=8 X

16 Legacy Methods By Contrast Assume Independence Of The Dimensions Of Risk Consider again our two, random variables X and Y that consist of risk observations for some project or program with the derived variance-covariance matrix In Euclidean Measure the distance between two points in XY-plane depends on the inverse of the Identity matrix Y 0 =(,) X-Y = (,)-(,)=(0,) Inv(Identity)= 0 0 = 0 0 Y X 0 =(,) Distance = (X-Y) 0 (X-Y)` 0 = (0,) 0 0 (0,)`=9 X

The MRisk Metric Calculates Distance While Accounting For The Point To Point Relationship Mahalanobis D is a multidimensional version of a z-score.

17 The MRisk Metric Calculates Distance While Accounting For The Point To Point Relationship Mahalanobis D is a multidimensional version of a z-score. It measures the distance of a case from the centroid (multidimensional mean) of a distribution, given the covariance (multidimensional variance) of the distribution. A case is a multivariate outlier if the probability associated with its D is 0.00 or less. D follows a chi-square distribution with degrees of freedom equal to the number of variables included in the calculation. Mahalanobis' distance identifies observations which lie far away from the center of the data cloud, giving less weight to variables with large variances or to groups of highly correlated variables (Joliffe, 986). This distance has advantages to other distance measures like the Euclidean distance which ignores the covariance structure and thus treats all variables equally 6

Case In Point: Multiple Risks Consider a risk scoring session involving risks SMEs vote on the probability and consequence (,) for five events across

18 Case In Point: Multiple Risks Consider a risk scoring session involving risks SMEs vote on the probability and consequence (,) for five events across three dimensions: Performance, Cost, & Schedule The average score across SMEs for each event is recorded in the table below RiskID Like Cost Schd Tech 7

19 Using MRisk All The Risk Consequences Fit Onto One Scale RiskID Like Cost Schd Tech RiskID Like MRISK:C MRisk answers the question regarding highest project risk A (,,) is more consequential than a (,,) or a (,,) min (,,) max (,,) The current methods would have us focus our attention on the (,,) and the (,,) despite the fact that they are not the most consequential 8

20 MRisk Provides A Clear Picture Of The Risk Profile Regardless Of The Number Of Dimensions Involved RiskID Like Cost Schd Tech MRISK:C Thi 9

21 Traditional Multivariate Methods Like Euclidean Distance May Not Be As Clear Because They Don t Consider Relationships RiskID Like Cost Schd Tech Euclid:C Lumping on severity despite differences Possible collusion of extreme risks 0

22 MRISK Is Not The Same As An Average MRISK RiskID Like Cost Schd Tech MRISK:C Avg(C) min (,,) The Average max (,,) min (,,) max (,,) MRisk highlights and brings attention to outlier risks An average, by contrast, masks differences between risks An average, shrinks the risk scale making it hard to distinguish between risks

23 MRISK Easily Adapts To Confidence Intervals Case In Point: Multiple SMEs Take the same example as before with five risk events Now consider that nine SMEs voted to create each risk score Taking RiskID, we can capture the uncertainty associated with this risk The score from each SME is recorded in the table below RiskID Like Cost Schd Tech RiskID SME Like Cost Schd Tech Avg

24 The Variance Between SMEs Supplies Uncertainty For Each Risk Event A 9% confidence interval for RiskID: goes from. to. in the consequence field RiskID SME Like Cost Schd Tech MRISK Avg (.,.)

25 MRisk Deals With Several Shortcomings In Legacy Risk Analysis MRisk uses Engineering Judgment to help decision makers get a full picture of the risk portfolio Just because we cannot visualize risk in multiple dimensions doesn t mean the dimensions don t exist. We all realize that Risk Management is a multi-dimensional problem. MRISK is a multi-dimensional solution to this problem. MRisk does not seek to change Risk Management from its current practices and procedures. It just revolutionizes Risk Analysis. MRisk does not require any change to current data collection techniques for implementation MRisk takes the data from the current risk methods and allows for interpretation of risks through a multidimensional lens The use of Mahalanobis Distance as a measure of consequence takes into account the relationships that risk events have across dimensions, i.e. cost, schedule, etc Since we know cost relates to schedule, schedule relates to performance, performance relates to safety, etc MRisk is most appropriate for measuring risk as it emphasizes the relationships among risks to calculate distance

client user GUIDE 2011

client user GUIDE 2011 STEP ACTION Accessing Risk Register 1. Type https://www.scm rms.ca/riskregister/login.aspx 2. Click in the Username field on the Risk Register home page. 3. Type your Username and