A Safety Operating Characteristic

Transcription

1 A Safety Operating Characteristic An alternative NAS performance metric Presenter: Sherry Borener, Ph.D. FAA Office of Aviation Safety Aviation Safety Analytical Services (ASA300) NextGen System Safety Management Transformation

2 Introduction and Overview Context NextGen, System Safety Management Transformation (SSMT) Development, and Overall NextGen investment decisions and performance metrics Historical Perspective Mechanistic models of performance at odds with operational requirements. Predictions of future capacity and associated risk may be missing an important intermediate variable therefore estimates of safety impacts are ambiguous. Current, Near and mid-term (2017) requirements Daily air system management Portfolio assessment and investment sequencing Evaluation and feedback Long term requirements Forecasting, modeling system requirements R&D decision-making 2

3 Conceptual Baseline Risk Accident Risk Traffic Growth Risk profile as changes are implemented

4 Baseline Commercial Rate and Targets 4

5 Baseline GA Rate and Targets 5

6 CAST Rating Process Use a CAST-like process for evaluation 6

7 Mid-Air Collision Fatalities Injuries Damages Near Miss Response? OE TCAS RA PD Loss of Situational Awareness CRM Loss of Situational Awareness Mode Confusion CRM IMC High Traffic Demand Comm Failure TMA NEO Weather CPDLC Synthetic Vision 7

8 Limitations of Current Benefit Forecasting Methods Identification of specific mechanisms of benefit are limited A lack of a standard set of test cases to quantify benefit in safety analyses leads to ambiguity and potentially poor decisions. Analyses are post-hoc (with long lag times) so that they are not helpful in real-time or near-real time decision making.; corrective action will not do. Use the everyday observable data (with a small twist) to evaluate the risk of the system and apply it to predict system risk. The relationship between scheduled traffic, capacity and safety-events such as errors (operational errors or pilot deviations), incidents and accidents form the operating characteristic. 8

9 Integrated Model/Metrics Picture Human Performance Indicator (workload, sit Awareness, etc.) Close Proximity Indicator (Aircraft closing distances) System States Total System Performance Risk/Capacity Benefit Mechanisms Event/Tech Assessments Risk Reduction (CAST) Benefit/Cost 9

10 Component #1: Human Performance Indicator An indicator of how workload / risk will be affected due to the proposed changes in the system Could be estimated from historical experience Could also be the output of experimentation Results are similar to outputs of detail level models such as air midas 10

11 Apr 27, 2007 Delays 194 Departure Arrival 0 16 TMS 178 % Ops delayed = 16% Average delay = 98 minutes Metar CW GS GDP RABR -RA -RABR RABR RABR RABR BR BR -RABR HZ HZ BR BR BR BR LOW CIGS/WIND LOW CIGS ASH2680 COA121 COA29 LOF7878 Arr_Rate OAG_Arr Arr_Ct BR OAG_Dep Dep_Ct Z 8 Z 9 Z 10 Z 11 Z 12 Z 13 Z 14 Z Diversions (2) 15 Z 16 Z 17 Z 18 Z 19 Z 20 Z 21 Z 22 Z 23 Z 0 Z 1 Z 2 Z 3 Z 4 Z 5 Z 6 Z Airborne_Holding 7 Z The MITRE Corporatio

12 System Safety Operating Characteristic (SOC) Event Probability vs. Capacity by Operational Part Accidents per operation is important but misses part of the issue Accidents, incidents, errors vary by operational circumstances Accidents by capacity (called, effective, etc.) allows you to build a forecast. Accident rates vary significantly by operational part so SOCs should be calculated independently Rates also vary significantly by domain (airport, terminal area, enroute) May want further subdivision or groupings 12

13 Research Question: How well would the Safety Operating Characteristic perform as a metric? This metric could feed into many inputs: Observed data (like errors) should relate very well to precursors (like separation losses) Precursors should be predictable based upon system state observations Models using the CPI could relate precursors to the values of the CPI, and can then be used to predict errors Actual errors can be used to calibrate the prediction model. 13

14 Relationship of Capacity and Risk Risk Capacity is a function of throughput and delay C w C B Current good-weather delay Current good-weather operating point Current good-weather throughput with NextGen NextGen operating point A C N NextGen operating point B Number of operations 14

15 But what about specific risks? Limitations of the SOC are Only describes performance in the aggregate Difficult to decompose for tracing causality therefore difficult to regulate based upon metric Address limitations by explicitly modeling sub-components Air vehicle systems Air space control systems Describe and evaluate the implications of NextGen for these components. 15

16 An Example: Look at the transition probabilities from low to high risk states Accident λ ι λ ι TCAS Alerts Controller / Pilot Action PD / OE λ ι λ i λ ι High Risk Situation Contoller / Pilot Action λ ι λ ι Normal Operation Aircraft/Airspace Change λ ι 16

17 Example: Modeling error relationships to accidents Accident TCAS Alerts Contoller / Pilot Action PD / OE 17

18 Modeling Normal to High-Risk Transitions High Risk Situation Contoller / Pilot Action λ 2 λ 1 Normal Operation Aircraft/Airspace Change 18

19 Example: Observed and Implicit Events Accident TCAS Alerts Controller / Pilot Action PD / OE High Risk Situation Controller / Pilot Action 19

20 Can this be done Available data (next slide) Example Conclusions 20

21 Conclusions Our benefit analysis is cumbersome and difficult to apply We have no standard metric or test cases with which to evaluate alternative NextGen investments One step toward this is standardizing our evaluation framework risk with respect to flight frequency, and Using the observed behavior of the system to condition our forecasts. 21

22 Was the airport overscheduled? Actual AAR / Called Rate AAR/Call Exceeded Ratios