Reliability Too Important to Leave to the Experts?

Transcription

1 Reliability Too Important to Leave to the Experts? Tony Frisch Edward West Xtera Communications Vodafone

2 Presenter Profile Edward West works for Vodafone in the Submarine Systems Engineering department. Since joining the company in 2000, he has worked on various cable system implementation and upgrade projects. He has a degree in Physics from Imperial College, London and is a Chartered Engineer and Chartered Physicist. He has attended the SubOptic conferences since 2004 as an author and presenter. Photo Name: Edward West Title: Principal Engineer Edward.West@vodafone.com

3 Presenter Profile Tony started at BT's Research labs and moved to Alcatel Australia and got involved in testing submarine systems. A move to Bell Labs gave him experience in terminal design and troubleshooting, after which he went back to Alcatel France, where he worked in ASN's Technical Sales before moving to head Product Marketing. He is now SVP, Repeatered Solutions for Xtera Communications. Photo Name: Tony Frisch Title: SVP Repeatered Solutions Tony.Frisch@Xtera.com

4 Presentation Overview E 1. Why do we need spares? 2. How to calculate how many spares you need 3. Practical problems and solutions 4. Summary No complex mathematics good approximations instead Focus mainly on submarine terminal equipment

5 Different viewpoints E 1. Technical More spares Protection schemes 2. Financial Need to minimise cost 3. Legal Penalty clause for outage Service Level Agreements (SLAs)

6 Objective E Availability Cost Risk reduction not prevention

7 The key parameters E Failure rate (Failures In Time) 1 FIT 1 failure 1E9 hours Mean Time Between Failures MTBF 1 failure rate Mean Time To Replace MTTR 2-24 hours Repair Turnaround Time RTT days Objectives Unavailability U Fraction Availability A = 1 U

8 How units fail over time The "Bathtub Curve" E

9 Determining the FIT value T 1. From the component supplier Add the FIT values of all the components in a unit What about the reliability of construction? How good are the values? 2. By similarity Sometimes the only practical solution for new components 3. From field returns Tests how good the construction is in the correct environment Takes time to get good statistics

10 Measuring FIT value T Put a number [ Q test ] units on test for a period of time [ T ] See how many fail [ Q fail ] FIT rate is Q fail / Q test x T x 1E9 Number of years for 1 failure FIT Target 10 samples 50 samples 200 samples 10, , Need to accelerate the ageing process for low targets

11 FIT value at 95% Confidence 95% UCL Derived from observed failures; covers uncertainty Most likely / expected value 10 T Relative FIT Value 1 failure 2 failures 5 failures 10 failures Always bigger than most probable failure rate Risk of being too safe..?

12 Component FIT (Real example) T 15,000 10,000 At 95% UCL FIT 5,000 Most likely value "Expected" value 0 0 2,000,000 4,000,000 6,000,000 8,000,000 10,000,000 Deployed device-hours

13 Network, Link and Spares Availability E A B Outage (minutes/year) DWA SDH Mesh Availability % % %

14 Terminal Equipment / Repeaters E Terminal equipment 10 Circuit Packs 10,000 FIT Failure rate = 10x 1E failures / year 1.14 years MTBF Replacement: hours Repeaters 100 repeaters 4 pairs 10 FIT/amplifier pair Failure rate = 4x 100x 1E failures / year 28.5 years MTBF Repair: days Cable damage much more likely than repeater failure

15 Unavailability Example: for repeaters E 100 repeaters with 4 pairs and 10 FIT/amplifier pair Expect MTTR failures / year 4 days (only 1 day actual repair) Unavailability x 4 = days per year? Average (all pairs) = days per year = 8.64 minutes/year Unavailability is essentially an average Not all repair times are the same

16 Unavailability T Need to consider all the units in an A-B connection A 1 B 2 3 Availability Unavailability A = A 1 x A 2 x A 3 U = 1 A U U 1 + U 2 + U 3 [ > 1 A i.e. "safe" ]

17 Just one or two units really matter T WL1 WL2 WL10 O P M U X Wavelength units complex 5,000-10,000 FIT failures / year Important (also expensive) MUX relatively simple 50-1,000 FIT failures / year

18 BREAK Any Questions?

19 Defining Number of Spares E Sparing is often a contentious issue during tendering / contract forming Each Supplier and Purchaser has its own preferred method Consistent sparing approach needed to compare "like with like" during Evaluation Purchaser should specify exactly how Tenderers should dimension their spares BUT also allow additional proposals which are better

20 Factors Affecting Spares Quantities: Modelling Traffic Impact Card categories: Non-traffic affecting cards Card failures where there is redundancy Card failures causing traffic outage for some traffic Card failures causing traffic outage for all traffic System availability can be achieved with lower spares availability. Don t necessarily need % availability of spares E

21 Doing The Sums Terminal Equipment E How to specify spares required? Needs to be easily understandable Needs to be demonstrable Several possible methods

22 Method 0: "No Spares" E Cheapest approach most risky Long outage associated with any failure Appropriate if there is redundancy Equipment protection usually protects just some units Network protection protects all units expensive Minimises outage times Effectively a different type of sparing

23 Method 1: "One Spare of Everything" E Simplest approach Does not take into consideration different working quantities Does not take into consideration different failure rates of different units

24 Method 2: "10% of Working Units" E Next simplest approach Crudely takes into account different working quantities Does not take into consideration different failure rates of different units

25 Method 3: Confidence Level (1) E Having enough spares in each station to meet a confidence level (e.g. 95%, 99%). That is, the probability of not running out of spares (having more failures within the RTT than you have spares) Use Cumulative Poisson formula to calculate probability of more failures than the number of spares PFAIL = POISSON(events, mean, TRUE) [ in Excel ] Number of Spares Expected number of failures before failed unit is returned Number of spares can be reduced by shorter RTT & MTTR

26 Method 3: Confidence Level (2) E Example Card FIT Working Qty Spares Qty Confidence Level MUX 3, % MUX 3, % XPD 10, % XPD 10, % XPD 10, %

27 Method 4: MTTR (1) E Base number of spares on Effective MTTR which includes time to: 1. notice and diagnose problem 2. find/insert spare card a. Local spare or b. From central spares depot (pool) or c. From repair if no spares are available

28 Method 4: MTTR (2) E Relate sparing levels to Effective MTTR, reflecting all contributions to repair times 99% spare availability = 1% chance of running out of spares => 0.01 x 90 days x 24 hours = 21.6 hours contribution to MTTR approximately Effective MTTR is missing link between spares and traffic Difficult to calculate and demonstrate

29 State transition diagram T Major states represent number of units in repair Number of units in repair Spare a No spare 3 4 Example with 1 spare

30 Starting condition T Number of units in repair Spare a No spare 3 4

31 Failure T Number of units in repair a No spare 3 4 Outage until failed unit is replaced Time depends on replacement speed Operator

32 Replacement T Number of units in repair Spare a No spare 3 4 Remain in State 1 until unit is repaired and returned Time depends on repair speed Supplier return speed Shipping & Customs

33 Failure with no spare T Number of units in repair Spare a No spare 3 4 Remain in State 2 until unit is repaired and returned Long outage until unit is repaired and returned

34 Second failure with no spare T Number of units in repair Spare a No spare 3 4 Remain in State 3 until unit is repaired and returned 2 units affected

35 How to analyse T 1. Monte Carlo Simulation Computer-randomised failures over a large number of test cycles No standard packages, so needs an expert; hard to verify 2. Calculation of probabilities Using Poisson, Binomial or Markov steady-state Easy, but needs care Can be done on a spreadsheet easier to verify

36 Monte Carlo Simulation T Start in State 0 and consider a day (say) at a time Randomly decide if a unit fails Determine if there is outage (is there a spare?) Check if a unit being repaired has returned Repeat many times Count outages, number of days without spares

37 Steady-state Markov T Calculates probability of states 0, 1, 2 a P(a) P(0) P(1) P(2) P(3) P(4) In the steady state transitions in and out must balance (very slightly pessimistic) Calculate Outage from P(a), P(2), P(3)

38 Methods compared T Units 5,000 FIT 4 hour MTTR 50 days RTT P(j) Simulation Sim. average Markov j

39 Reducing outage More spares, Shorter MTTR E Outage (minutes/year) MTTR 4 hours hour Number of spares No value in having too many spares

40 Reducing outage More spares, Shorter Return times E TTR Outage (minutes/year) Number of spares Worthwhile if the return time is certain

41 BREAK Any Questions?

42 Practical Problems and Solutions Theory versus real life E Problem Error in repair Damage during shipment "Out of box" failure Human error in station, e.g. delay in returning faulty unit Spares too expensive Outage too high Excessive failures Comment Clear description of fault Robust shipping containers Test spares Return without delay Pooling Protection Legal

43 Reducing outage Using a pool of spares E Pool of spares reduces return time if a station runs out of spares One country or multiple countries? Timelines dependent on customs clearance Who owns the pool? Supplier to hold spares pool? S S S

44 Reducing outage Using a pool of spares E Outage (minutes/year) Days to move pool spare 5 2 No pool spare 1 pool spare Number of spares Cost effective when possible

45 Reducing outage Using a protection switch T No outage for failure when spare is available Need to consider: Failures affecting one channel Failures affecting ALL channels Replacement process N x N Matrix 100G 100G

46 Reducing outage Using a protection switch T Outage (minutes/year) Unprotected 10.0 Protected Number of spares Outage dominated by switch and common failures

47 Hot or cold Spares? E When is a spare not a spare? In a "protected" system, should the protecting card be considered as a hot spare? Powered spares rack Hot or cold? For testing only, or to monitor spares? Powered spares slot in working rack A more compact / cheaper solution Hot units can fail: cold ones shouldn't but how will you know?

48 Failure rate higher than expected? E Example four sites each with 16 Units at 5,000 FIT Expected failures Station failures in first year 0.7 per year per site 3 in one site only Failures One station % % % 3 2.8% 96.6% failures in one site seems excessive

49 Failure rate higher than expected? A different viewpoint T Example four sites each with 16 Units at 5,000 FIT Expected failures Station failures in first year 2.8 per year overall 3 overall Failures One station All stations % 6.1% % 17.0% % 23.8% 3 2.8% 22.3% 46.9% 3 failures overall seems OK?

50 Distribution of failures Assuming 3 failures T In 1 site combinations In 2 sites combinations In 3 sites combinations

51 Outage Assuming 3 failures T Assume 2, 3 and 4 hours for the replacement times 10 hours total outage With 2 spares expected 30 minutes per year However, there are 32 circuits 10 x 60 / 32 = 19 minutes / year Outage is within prediction

52 Summary E Objectives Obligations Purchaser Failures cost money Too many spares cost money Consider whole network Define methodology to compare offers Maintain MTTR targets Return failed units without delay Supplier Failures cost money Selling spares is a business opportunity Offer innovative solutions Maintain RTT targets Good packaging & carriage Improvements Monitor Supplier performance Test returned units Consider pool of spares Consider N+1 protection Monitor FIT rates Innovative protection schemes

53