Quantitative Risk Modelling, Calibration and Continuous Improvement CK UMACHI RISK MANAGEMENT ENGINEER - TIMP PACIFIC GAS & ELECTRIC

Transcription

1 Quantitative Risk Modelling, Calibration and Continuous Improvement CK UMACHI RISK MANAGEMENT ENGINEER - TIMP PACIFIC GAS & ELECTRIC

2 Agenda Relative vs Quantitative Risk Models PG&E s Risk Model History PG&E s Risk Methodology Past Risk Model Results and Takeaways Calibration of the Models Continuous Improvement and Future Plans Challenges 2

3 Relative vs Quantitative Risk Models Relative/Qualitative Model Assigns a unit-less scoring to each given asset Scales and magnitudes of the scores are somewhat arbitrary Scores are able to be ranked Quantitative Model Calculates a value with units for a given asset Magnitudes of the scores are representative of the risk level Scores are able to be ranked Risks can be compared across threat types, asset types, etc Allows for portfolio level decision making 3

4 Pitfalls of Quantitative Models Can t force a relative model into a quantitative framework and expect good results Need high quality data All models are wrong, some models are useful 4

5 Agenda Relative vs Quantitative Risk Models PG&E s Risk Model History PG&E s Risk Methodology Past Risk Model Results and Takeaways Calibration of the Models Continuous Improvement and Future Plans Challenges 5

6 PG&E s Transmission Pipeline Risk Model History Relative Model 2016 Semi Quantitative Model 2017 Calibration Relative Risk Model introduced in 2001 Relative/Qualitative Scoring for Likelihood of Failure (LOF) and Consequence of Failure (COF) for each pipeline segment by threat SME input utilized to derive most scores 6

7 PG&E s Transmission Pipeline Risk Model History Relative Model 2016 Semi Quantitative Model 2017 Calibration Semi-Quantitative Risk Model first implemented in 2016 Quantitative Likelihood of Failure (ruptures per mile per year) and Qualitative Consequence of Failure (relative scale) Factors from the relative model utilized with a quantitative model framework for most threats Fault tree methodology from industry research utilized for the Third Party Damage model 7

8 PG&E s Transmission Pipeline Risk Model History Relative Model 2016 Semi Quantitative Model 2017 Calibration Initial Calibration of PG&E Risk Models performed in 2017 Required new or updated models in 2 cases Scoring ranges updated Results were much more in line with the historic rupture rates 8

9 Agenda Relative vs Quantitative Risk Models PG&E s Risk Model History PG&E s Risk Methodology Past Risk Model Results and Takeaways Calibration of the Models Continuous Improvement and Future Plans Challenges 9

10 PG&E s Semi-Quantitative Risk Methodology Risk = LOF COF Likelihood of Failure (LOF) - Quantitative values of rupture frequencies Calculated for each of the 9 threat categories from B31.8S Consequence of Failure (COF) - Qualitative values of consequences of ruptures 10

11 PG&E s Semi-Quantitative Risk Methodology - LOF LOF = Baseline Rupture Rate AF Baseline Rupture Rate is based on industry, company, or other data The adjustment factor shifts the rupture rate of a given segment up or down based on a failure likelihood score between 0-10 Between 2 orders of magnitude lower or higher than the baseline rupture rate 11

12 PG&E s Semi-Quantitative Risk Methodology - LOF The failure likelihood score is based on threat specific equations These equations are used to translate the threat specific factors into a score from

13 PG&E s Semi-Quantitative Risk Methodology - COF The COF model is based on 3 classes of consequence 13

14 Agenda Relative vs Quantitative Risk Models PG&E s Risk Model History PG&E s Risk Methodology Past Risk Model Results and Takeaways Calibration of the Models Continuous Improvement and Future Plans Challenges 14

15 Ruptures per Year 2016 Risk Model Results 2016 Results vs Historical PG&E Rupture Rate Sum of 2016 Risk Output Rupt/yr Sum of PG&E Rupt/yr ( ) TPD EC WROF IC C IO M EQ SCC Note: The values used are for illustration purposes only and not based on actual PG&E data. 15

16 Ruptures per Year 2016 Risk Model Outputs 2016 Results vs Historical PG&E Rupture Rate Sum of 2016 Risk Output Rupt/yr Sum of PG&E Rupt/yr ( ) TPD EC WROF IC C IO M EQ SCC Note: The values used are for illustration purposes only and not based on actual PG&E data. 16

17 Log(PG&E Model Results/Industry) 2016 Risk Model Outputs Results vs Industry Rupture Rate C EC EQ IC IO M SCC TPD WROF Note: The values used are for illustration purposes only and not based on actual PG&E data. 17

18 2016 Risk Models Results Takeaways The 2016 rupture frequency results were clearly overestimated Thankfully, we didn t have as many ruptures in 2017 as the model predicted Were the results completely invalid? No, the overall results didn t discount the relative rankings of the scores Models needed to be calibrated We needed to figure out to analyze the results and improve the models 18

19 Agenda Relative vs Quantitative Risk Models PG&E s Risk Model History PG&E s Risk Methodology Past Risk Model Results and Takeaways Calibration of the Models Continuous Improvement and Future Plans Challenges 19

20 Calibrating The Models Evaluate Comparison of Model Results vs Historic Failure Rates Evaluated Results based on Good Assumptions Evaluated Risk Model Factors with Sensitivity Analyses Propose Changes Looked for systematic changes first Need to understand model mechanics Re-evaluate Repeated the evaluate stage 20

21 Assumptions Assumption 1 No historic failures If no failure events have occurred for a given threat, it may not make sense for the model to produce a higher LOF than the industry Assumption 2 Historic failures Model outputs should be in line with historic failure rates if nothing has changed 21

22 Sensitivity Analysis (Tornado Diagrams) Sensitivity Analysis Determines how variations in inputs affect the outputs of a model Tornado Diagrams bar charts which visually represent a sensitivity analysis Why is this important? Isolates most impactful factors Identifies non-impactful factors Allows for comparisons between SME expectations and model the Factor 1 Factor 2 Factor 3 Factor 4 Factor 5 Factor 6 Factor 7 Factor 8 Factor 9 Factor 10 Factor 11 Factor 12 Factor 13 Factor 14 Factor 15 Factor 16 Factor 17 Risk Factor Sensitivity results 5.0E E E-04 LOF (ruptures/mi-yr) 22

23 2017 Model Changes Re-scaled the scoring ranges 100x higher and lower than baseline ranges were overly inflated Created new models where calibration wasn t enough Added, removed and moved factors to better represent the threats with regards to rupture Corrected issues with the model math 23

24 Ruptures per Year Effects of Calibration on PG&E s Risk Model Outputs 2016 Results vs 2017 Results vs Historical PG&E Rupture Rate Sum of 2016 Risk Output Rupt/yr Sum of 2017 Risk Output Rupt/yr Sum of PG&E Rupt/yr ( ) TPD IC EC WROF C SCC IO M EQ Note: The values used are for illustration purposes only and not based on actual PG&E data. 24

25 Log(PG&E Model Results/Industry) Effects of Calibration on PG&E s Risk Model Outputs 3 PG&E Results vs Industry Historical Rupture Rates 2 1 Sum of Log 2016 Results/Industry 0 C EC EQ IC IO M SCC TPD WROF Sum of Log 2017 Results/Industry Note: The values used are for illustration purposes only and not based on actual PG&E data. 25

26 Agenda Relative vs Quantitative Risk Models PG&E s Risk Model History PG&E s Risk Methodology Past Risk Model Results and Takeaways Calibration of the Models Continuous Improvement and Future Plans Challenges 26

27 Continuous Improvement 2016 calibration effort focused on higher level changes Want to focus on more data driven enhancements to the models and factors Utilize data sets to calibrate specific factors Steering committee meetings to solicit feedback from SMEs Hundreds of recommendations Model restructuring New data sets Process enhancements 27

28 Future Plans Transition to a fully quantitative risk model Quantitative Consequence of Failure Probabilistic Models Probability of exceedance (POE) calculations implemented in 2017 Quantified likelihood of rupture for external and internal corrosion defects based on In-Line Inspection data Explored in-house capability to create probabilistic model for external corrosion with limited success Will be exploring available technologies for probabilistic modelling 28

29 Future Plans Risk Standardization and Optimization Work Developing framework for easily displaying, rolling up, and comparing risks Quantifying risk reduction achieved from mitigations Determine optimal mitigation strategies Different departments are aligning on standardization of risk concepts 29

30 Risk Standardization and Optimization Framework 30

31 Agenda Relative vs Quantitative Risk Models PG&E s Risk Model History PG&E s Risk Methodology Past Risk Model Results and Takeaways Calibration of the Models Continuous Improvement and Future Plans Challenges 31

32 Challenges Resources Analytical personnel (data scientists, analysts) Tons of work to improve models Data Data availability Data quality Data interpretation Modelling Using simple models to consider highly complex scenarios Uncertainty needs to be accounted for 32

33 Thank you! 33