@ - Presentation Caveat

Transcription

1 @ - Presentation Caveat The following presentation was made by Marv Nuss of Nuss Sustainment Solutions at the 2013 Aircraft Airworthiness and Sustainment Conference Australia. The presentation title is: An Outsider s Perspective on FAA s Aircraft Safety Risk Assessment & Risk Management Approach. Mr. Nuss describes the USA Federal Aviation Administration s (FAA) approach to risk assessment and risk management for aircraft continued operational safety. Mr. Nuss, a retired FAA certification engineer, explains the FAA s Transport Airplane Risk Assessment Methodology (TARAM) published in FAA document PS-ANM He uses the 2013 Boeing 787 lithium-ion battery failures as an example. The presentation contains facts from publicly available documents, including published USA s National Transportation Safety Board (NTSB) reports and newspaper reports. Mr. Nuss received no information from the FAA or Boeing for the presentation. The assumptions made for the example risk assessment using the TARAM method are those of Mr. Nuss, and are based only on publically available information. Mr. Nuss believes his assumptions to be realistic and a good example of the TARAM method. However, it is completely an outsider s perspective. Mr. Nuss believes quantitative risk assessment is an effective means to manage risk. Concepts similar to those explained in the presentation provide valuable tools for continued operational safety managers whether a regulator, manufacturer, or operator. The presentation is intended to demonstrate how such a risk assessment works, not as a critique of either the FAA or Boeing. The FAA s risk assessment and management methods continue to evolve. Mr. Nuss understands that the FAA s TARAM document used for the example in this presentation is likely to be revised with updated policy at some point in the future. Mr. Nuss asks the reader to view the presentation with the understanding of his outsider s perspective. He also hopes the reader finds it informative and provides a better understanding of quantitative risk assessment and risk management.

2 An Outsider s Perspective on FAA s Aircraft Safety Risk Assessment & Risk Management Approach 2013 Aircraft Airworthiness & Sustainment Conference (Australia) Brisbane, QLD July 23-25, 2013 Marv Nuss marv.nuss@marvnuss.com

3 An Outsider s Perspective on FAA s Aircraft Safety Risk Assessment & Risk Management Approach Background FAA approach Definitions Case study 1

4 RA&RM Background Managing risk is the bottom line task for any enterprise Risk, one definition: the existence of a potential for an undesirable loss Undesirable loss could be: Life Health Money Time Customers In the aerospace industry, these risks can be competing Safety risk vs. mission readiness risk Safety risk vs. design risk Safety risk vs. financial risk Financial risk vs. mission readiness risk RA&RM: Risk Assessment and Risk Management 2

5 RA&RM Background In order to manage risk it must first be quantified Risk assessment is a means of quantifying risk Useful data are necessary for sensible risk assessment Computer computational speed enables easier risk assessments Ability to collect and store data Ability to analyze large amounts of data Increased pressure to make processes more efficient puts increased pressure on understanding associated risks 3

6 Probability, per hour Big Picture Perspective on Life Risk 1.0E-04 Professional Rodeo (Bull, Bronc) Motorcycle Individual Death Probability 1.0E-05 Female Probability of Death - All Causes 1.0E-06 Male Probability of Death - All Causes Passenger Vehicle Individual Death Probability TARAM Individual Risk Threshold 1.0E-07 Average Accidental Death Probability Passenger Bus Individual Death Probability Drowning in Bathtub Individual Death Probability Commercial Airplane (part 121) Individual Death Probability 1.0E Age (years) An Outsider s Perspective on FAA s Aircraft Safety Risk Assessment & Risk Management Ref: Approach FAA TARAM Handbook 4

7 FAA s Disciplined Approach to Risk Management FAA established its Safety Risk Management Policy in 1998: The FAA shall use a formal, disciplined, and documented decision-making process to address safety risks in relation to high-consequence decisions affecting the complete life cycle. (FAA Order , revised to A in 2012) FAA Aviation Safety organization followed with its Safety Management System (SMS) FAA Aircraft Certification Service SMS includes specific policies regarding risk assessment and management Monitor Safety/Analyze Data (MSAD) is FAA s Aircraft Certification Service s process to manage risk: designed to promote data-driven, risk-based continued operational safety decision-making. The MSAD process is documented in FAA Order A (2012) 5

8 FAA MSAD Risk Management Process MSAD process relies heavily in-service data. Service difficulty reporting (SDR) is the major source of in-service data. Ref: FAA Order A 6

9 FAA MSAD Risk Management Process MSAD is an attempt to standardize the safety risk associated with any failure Mechanical systems Electrical systems Engine and fuel systems Structure Each directorate developed specific process for risk assessment and management Transport Directorate: TARAM -- Transport Airplane Risk Assessment Methodology, published Nov (PS-ANM-25-05) Small Airplane Directorate: SARA Small Airplane Risk Analysis Rotorcraft and Engine/Propeller Directorates have similar processes 7

10 FAA Risk Analysis Specification Process Statistical Exposure (Time, Number of Flights) Frequency of Occurrence (Failure Rate, Event Rate) (Number of Occurrences per Hour, per Flight) Probability (Exposure x Frequency of Occurrence) Conditional Probability (Event to Severity) Severity (Fatal Accident) Quantitative Risk (Expected Probability of an Fatal Accident) Measurable 8

11 FAA Safety Risk Definitions Severity definitions Catastrophic: Multiple fatalities (or fatality to all on board) usually with the loss of aircraft Hazardous: Multiple serious injuries; fatal injury to a relatively small number of persons (one or two) or a hull loss without fatalities Major: Physical distress or injuries to persons and/or substantial damage to aircraft Minor: Physical discomfort to persons and/or slight damage to aircraft Minimal: Negligible safety effect Likelihood definitions Extremely improbable: So unlikely that it is not expected to occur, but it is not impossible Extremely remote: Expected to occur rarely Remote: Expected to occur infrequently Probable: Expected to occur often Frequent: Expected to occur routinely Ref: FAA Order A 9

12 FAA Risk Assessment and Management Definitions Causes: Underlying circumstances, occurrences, and/or failures that contribute, or could contribute, directly or indirectly, to an event. Corrected risk: Residual risk that remains after corrective action is taken. When highly effective corrective action is taken, residual risk is considered to be zero. Corrective action: Any action to mitigate a safety issue. Fleet: Aircraft, engine or propeller products of a type currently in service affected by a certain safety issue. Hazard: Any existing or potential condition that can lead to injury, illness or death to people; damage to or loss of a system, equipment or property or damage to the environment. A hazard is a condition that is a prerequisite to an accident or incident. Probability: Ratio of the number of actual occurrences to the number of possible occurrences. Risk: Expression of the severity and probability of an undesired event. Severity: The consequence or impact of a hazard in terms of degree or loss or harm. Uncorrected risk: Risk that accumulates over time in the affected fleet if no corrective action is taken for a certain safety issue. Ref: FAA Order A 10

13 FAA Risk Assessment Definitions Conditional Probability (CP): the probability that an unsafe outcome, for which an injury ratio is known, will result from a particular condition under study. The conditional probability is the product of the individual conditional probabilities for all of the conditions that must occur, after the condition under study, to result in the defined unsafe outcome. Defect Airplanes(s) (DA): the predicted number of airplanes that would have the subject failure if the condition under study is left undetected during the timeframe being analyzed. Exposed Occupants (EO): the average number of persons expected to be exposed to fatal injury during an unsafe outcome or condition. Frequency of Occurrence (F): the rate at which the condition under study is expected to manifest itself within the affected fleet or sub fleet. For non-constant failure rates, such as wear-out failures, Weibull and log-normal analysis techniques are helpful in determining the distribution of failures over time. Hazard Function (h(t)): the instantaneous failure rate of a unit. Hazard function is analogous to the frequency of occurrence. Ref: PS-ANM FAA TARAM Handbook 11

14 FAA Risk Assessment Definitions (con t) Injury Ratio (IR): the average single-event probability that those exposed to a particular condition or outcome will suffer fatal injury. Utilization (U): the airplane s flight hours or flight cycles per defined-unit time period (e.g., per day). Not Detected (ND): the probability that an occurrence of a defect will not be detected before the defect leads to an unsafe condition or outcome. ND is a conditional probability, but it is defined separately because of its importance in certain wear-out issues, such as structural fatigue. Number of Aircraft (Σ): the number of airplanes in the affected fleet during the time period under study. Time Period (T): the time period over which risk is calculated. See Paragraph 4.6 for specific guidance on time period determination. Ref: PS-ANM FAA TARAM Handbook 12

15 FREQUENCY of OCCURRENCE (Failure Rate / Event Rate per Hour / Flight / Cycle) Components of Fleet Risk (Total Uncorrected Fleet Risk) FLEET EXPOSURE (Time, Flights, Cycles ) OUTCOME SEVERITY RATE (Injury Ratio) -OR- (90-Day and Control Program Fleet Risk) EXPOSED OCCUPANTS (Passengers and Crew) OUTCOME SEVERITY RATE (Injury Ratio) NUMBER OF OCCURRENCES (Fleet Exposure and Frequency of Occurrence) CONDITIONAL PROBABILITY (Probability of Outcome Given the Occurrence) SEVERITY (Weighted Events or Number of Expected Fatalities per Outcome) Risk (Probability of Fatality over Time or Number of Fatalities over Time ) Components of Individual Risk FREQUENCY of OCCURRENCE (Failure Rate / Event Rate per Hour / Flight / Cycle) CONDITIONAL PROBABILITY (Probability of Outcome Given the Occurrence) OUTCOME SEVERITY RATE (Injury Ratio) Risk (Individual Probability of Fatal Injury per Flight Hour) An Outsider s Perspective on FAA s Aircraft Safety Risk Assessment & Risk Management Ref: Approach FAA TARAM Handbook 13

16 TARAM Risk Assessment Equations Fleet exposure: U x T x Σ Predicted number of Occurrences: U x T x Σ x F Severity: S = IR Fleet Risk: R = (U x T x Σ x F) x CP x S Individual risk: R = F x CP x S Individual risk: Probability of individual fatal injury per flight hour Variants of these equations are used to determine: Uncorrected and corrected risk (constant rate or wear-out) Short term risk U: Utilization T: Time Period Σ: Number of Aircraft F: Frequency of Occurrence S: Severity IR: Injury Ratio R: Risk CP: Conditional Probability TARAM: Transport Airplane risk Assessment Methodology An Outsider s Perspective on FAA s Aircraft Safety Ref: Risk PS-ANM Assessment & Risk FAA Management TARAM Handbook Approach 14

17 Risk Assessment Case Study: B-787 Battery Fires Was the FAA justified in grounding the fleet? Information sources: NTSB Interim Factual Report for NTSB No. DCA13IA037, March 7, 2013 FAA special conditions No SC, Federal Register October 11, 2007, page Wall Street Journal (WSJ), various dates Aerospace An Outsider s & Perspective Defense News on FAA s (ASDNews), Aircraft Safety various Risk Assessment dates & Risk Management Approach 15

18 B-787 Battery Fires Information Timeline Jan. 7, 2013: APU battery event in Boston (JAL) Jan. 16, 2013: main battery event in-flight in Japan (ANA) Jan. 16, 2013: FAA issued emergency AD that grounds the fleet Pertinent incident information JAL: heavy smoke and fire coming from front of battery case (3 flames) JAL: intense smoke in cabin ANA: burning smell and smoke in the cabin, battery malfunction similar to JAL Fleet history as of Jan. 16, airplanes in service Total accumulation: 51,662 hours, cycles Certification information Battery vent and/or smoke (without fire) classified Hazardous (Extremely remote, 1 x 10-7 /hour) Battery vents smoke/fire classified Catastrophic (Extremely improbable, 1 x 10-9 /hour) 16

19 JAL JA829J APU Battery Fire Photos Forward face of the battery case APU installation location with battery removed Ref: NTSB #DCA13IA037 Interim Factual Report, 3/7/13 Side face of the battery case Battery case with sides folded down 17

20 ANA Main Battery Fire Photo View into the battery case Ref: WSJ 18

21 TARAM Risk Assessment Frequency of Occurrence (F) = Occurrences/Fleet Exposure: F = 2/52,000 hrs. 4x10-5 /hr. (400 times greater than cert (1x10-7 )) A comparison example: Fleet of 500 x 50 hrs/week x 50 weeks 1.25M hrs./year (.125x10-7 ) Certificated expectation of hazardous battery vent and/or smoke (without fire) no more often than once/ 8 years for this example TARAM: Transport Airplane risk Assessment Methodology 19

22 TARAM Risk Assessment Conditional Probability (CP): the probability that an unsafe outcome, for which an injury ratio is known, will result from a particular condition under study. Per TARAM guidance, assume next failure results in a more severe outcome First two battery failures could be classed as Major or Minor Assume next failure either Catastrophic or Hazardous CP =.333 (2 failures + next failure (with assumed higher severity) = 3 occurrences, one of which is an unsafe outcome) Severity (S): The consequence or impact of a hazard in terms of degree or loss or harm. Severity definitions Catastrophic: Multiple fatalities (or fatality to all on board) usually with the loss of aircraft Hazardous: Multiple serious injuries; fatal injury to a relatively small number of persons (one or two) or a hull loss without fatalities Major: Physical distress or injuries to persons and/or substantial damage to aircraft Minor: Physical discomfort to persons and/or slight damage to aircraft Minimal: Negligible safety effect An Outsider s Perspective on FAA s Aircraft Safety Risk Assessment Ref: & FAA Risk TARAM Management Handbook, Approach FAA Order A 20

23 TARAM Risk Assessment Injury Ratio (IR): the average single-event probability that those exposed to a particular condition or outcome will suffer fatal injury. (IR = S) Per TARAM guidance example, IR = 0.16 for in-flight fire Individual risk: R = F x CP x S R = (4 x 10-5 /hr. ) x x 0.16 R 2 x 10-6 /hr. TARAM guideline for normally accepted individual risk is R 1 x 10-7 /hr. For the scenario of a battery failure causing an individual fatality due to a fire, the risk is 20 times higher than the TARAM guideline. This would justify the FAA s AD. Ref: PS-ANM FAA TARAM Handbook 21

24 TARAM Risk Assessment Fleet risk: R = (U x T x Σ x F) x CP x S R = (5 x 365 x 20 x 1000 x 4x10-5 ) x x Hrs./day Days/yr. Yrs. No. A/C Freq. CP S = IR TARAM guideline for normally accepted fleet risk is R 0.02 Note: Hrs./day and years are probably low estimates Ref: PS-ANM FAA TARAM Handbook 22

25 TARAM Risk Assessment Fleet risk: R = (U x T x Σ x F) x CP x S R = (5 x 365 x 20 x 1000 x 4x10-5 ) x x Hrs./day Days/yr. Yrs. No. A/C Freq. CP S = IR TARAM guideline for normally accepted fleet risk is R day fleet risk: R = (U 90 x T x Σ 90 x F) x CP x IR x EO R = (5 x 90 x 50 x 4x10-5 ) x x 0.16 x Hrs./day Days No. A/C Freq. CP IR Exposed Occ s. TARAM guideline for normally accepted 90 day fleet risk is R 0.5 Note: Hrs./day probably a low estimate Ref: PS-ANM FAA TARAM Handbook 23

26 TARAM Risk Assessment It appears that risk of battery malfunctions is an unsafe condition Individual per hour risk (20x FAA normally accepted risk) Fleet risk (4000x FAA normally accepted risk) 90 day fleet risk (10x FAA normally accepted risk) Assumptions appear to be reasonable However, injury ratio (IR) may be too high for the 787 scenario One order of magnitude less would put 90 day risk within acceptable range From an outsider s perspective, it appears the FAA was correct issuing its AD. However, additional data could substantiate a reduced injury ratio that would justify continuing operations for a short time to allow design and installation of a mitigating risk control. 24

27 Comparison of Old and New 787 Battery Redesign WSJ photo Mitigating redesign features Increased space between cells Fireproof containment box Vent fumes outside Detailed pre-flight data review Repeated in-flight status checks Original Ref: WSJ 2/21/13 An Outsider s Perspective on FAA s Aircraft Safety Risk Assessment NTSB & Risk photo Management Approach 25

28 New 787 Battery Reduces Risk FAA issued AD on April 26, 2013 Mandates installation of new batteries per Boeing Service Bulletin Mitigating redesign features should sufficiently reduce risk Increased space between cells Fireproof containment box Vent fumes outside Detailed pre-flight data review Repeated in-flight status checks If each feature reduces risk by factor or 10: Fleet risk goes from 80 to Reduces risk to 25 times less than TARAM normally accepted fleet risk (0.02) 26

29 An Outsider s Perspective on FAA s Aircraft Safety Risk Assessment & Risk Management Approach Summary RA&RM is a part of life Quantitative RA&RM is should be a skill of every aircraft engineer FAA has a rational approach to RA&RM The FAA AD to ground the B-787 appears justified Boeing s mitigations appear to lower risk significantly 27

30 An Outsider s Perspective on FAA s Aircraft Safety Risk Assessment & Risk Management Approach Questions? Perspectives? Thanks for your attention! Marv Nuss marv.nuss@marvnuss.com