Guard-banding Methods-An Overview

Transcription

1 AdMet 2012 Paper No. UM 001 Guard-banding Methods-An Overview Swanand Rishi ETDC, STQC Directorate, Department of IT, Govt. of India, Agriculture College Campus, Shivajinagar, Pune ABSTRACT : Guard-banding is a method of setting test limits lower than Specifi cation limits (SL) in order to optimise False-Accept & False-Reject risks. A guard-band is the offset from the SL to the acceptance test limit (TL) that is used for the pass/fail decision. It is used to reduce the False-Accept Risk, which increases the confi dence that a device is really in-specifi cation by factoring in the measurement errors. This involves the risk of false-accept as well as false-reject, owing to various factors like specifi cation limits, TUR, measurement uncertainty & a-priory probability of the device being in-tolerance. This paper presents various prevalent guard-banding methods in order to manage such risks in calibration. INTRODUCTION : Keywords : Guard-band, TUR, Uncertainty, Specifi cation Limit & Tolerance Limits In a near-ideal calibration scenario, one would expect reading of Device Under Calibration (DUC) closely matching the standard s value, indicating high process capability. The Specifi cation Limits (SL) will be much wider under such conditions. With consistent improvement in process capability, manufacturer would tighten the specs & shorten the SL band. In such a real-life scenario, the scatter of readings is not always close to standard value; many values found lying close to the specifi cation limits. This is particularly the case after a few years of use & when an item is subjected to recalibration. The ISO IEC 17025:2005 [1] calls for incorporation of uncertainty while making compliance statement. As such, the readings which are close to SL are likely to fall outside SL, leading to non-compliance. The False-Accept risk is the probability that a device which is actually out-of-tolerance is accepted due to measurement process error or the uncertainty. A False-Accept occurs when the UUT bias is out of tolerance, but deviation is not. It is also called as Consumer s risk or 1 Probability of False Acceptance (PFA). ISO/ IEC is silent on maximum level of false accept risk. However, Clause 5.3 of ANSI/ NCSL [2] specifi es PFA or Consumer s risk to be not more than 2%. There are two types of False-Accept risks- unconditional & conditional risks. An Unconditional False-Accept Risk is the average risk for a population of calibrated devices. It is an appropriate statistic for managing a large no. of instruments - typically a manufacturing scenario. On the other hand, conditional False- Accept Risk is appropriate when dealing with a specifi c instrument - typically a recalibration scenario. The False-Reject risk is the probability that a device which is actually in-tolerance is rejected due to measurement process error. The False- Reject risk is also called as Producer s risk or Probability of False-Rejection (PFR). This risk is evaluated in lieu of anticipated additional costs like re-calibration, adjustment or rectifi cation. One must note that increasing Guard-band for reducing False-Accept risk (Consumer s risk) disproportionately increases the False-Reject risk (Producer s risk) as seen from Fig. 1.

2 Table 1: False-Accept & False-Reject risk at different TURs at SL = 2σ Fig. 1 (a) : Producer Risk Probability of False Acceptance (PFA) could be altered by fi ne-tuning of calibration system control tools like- Measurement reliability Calibration intervals Calibration process uncertainty Calibration adjustments Guard-bands This paper deliberates on the last option. Before delving into various methods, some important terms need attention. Guard-band : It is nothing but range by which the specifi cation limit (SL) is reduced so that there is more confi dence in the Pass/ Fail decisions, which are taken based on tolerance limit (TL) rather than SL. Particularly employed when TURs are lower than 4:1, this is a safety margin for tightening an acceptance (pass) limit. Thus, Guard-band GB = SL - TL. It is also expressed as a multiple of SL, i.e. Fig. 1(b) : Consumer Risk This risk shows sharp rise at lower TURs. As there are cost implications, balance between the two risks is necessary. The selection of a Guard-band method & its decision rule is more a business decision for manufacturers whereas it is a matter of quality policy for the test & calibration labs. Table 1 shows some values from graph in Fig. 1 for specifi cation limit of 2σ (i.e. approx. 95% CL or In-tolerance probability & assumed normal distribution for UUT specifi cations as well as uncertainty). GB = K SL; K being 1. The Guard-band is usually set at a point equal to the specifi cation minus the uncertainty but is often adjusted to get the same confi dence that results at a TUR of 4:1 & acceptance limit set at the SL. Guard-bands are employed by manufacturers as well as calibration/test labs. The concept is illustrated in Fig. 2. In conditional False-Accept risk, generally larger guard bands are used, while smaller guard bands are employed in unconditional False-Accept risk. Note that Guard-band doesn t altogether avoid False- Accept or False-Reject risk, but increases confi dence in pass/fail decisions. 2

3 A-Priory Probability Distribution : The evaluation of PFA assumes a-priory probability that the device being calibrated is in-tolerance prior to actual calibration. Fig. 2: Guard-band (shown at Uncertainty) Specification Limits (SL): These are specifi ed by manufacturer either as rectangular, usually two-sided symmetrical specs, or at 95% confi dence level. It is stated in GUM [3] that, when a specifi cation is quoted for a given coverage probability, then a Normal (Gaussian) distribution can be assumed. The methods described in the paper are based on this assumption. Test Uncertainty Ratio (TUR) : A TUR is the ratio of DUC specifi cations to the measurement process uncertainty, both specifi ed at same confi dence level. A higher TUR reduces probability of wrong Pass-Fail decisions. A TUR of 10:1 is recommended in many standards & is in force for a couple of decades. The old US military standard MIL- STD-45662A [4] & old ANSI STD Z [5] recommend a minimum TUR of 4:1. Due to technological advancements & declining costs of precision equipment, a TUR of 4:1 is widely accepted nowadays. (Labs cannot afford to keep standards to maintain TUR of >4:1 for all workload/test points) The False- Accept risk & False-Reject risk become major concerns when the TUR is worse than 4:1, other factors remaining same. TUR has direct effect on the risk probability- the lower the TUR the higher the False-Accept risk as well as False-Reject risk. Thus a lower TUR is not desirable from both consumer s & producer s risk perspective. Measurement Uncertainty : It is the probable measurement process error, usually specifi ed at approx 95% confi dence level at a coverage factor k = 2 & is mandatory for accreditation per ISO/IEC Depending upon test-point reliability or reliability of population of UUTs, the PFA varies. Reliability fi gures of 80 to 95 % are common. The higher the reliability, the lower the PFA. VARIOUS GUARD-BANDING METHODS: ISO17025:2005, cl states: When statements of compliance are made, the uncertainty of measurement shall be taken into account. However, it provides no specifi c guidance for taking the measurement uncertainty into account when assigning Pass/Fail status nor does it specify maximum level of false accept risk. This allows different guard banding strategies to co-exist; some of the prevalent methods are presented here. Note that tables below are based on Specifi cation Limit (SL) normalised to 100 for ease of use & understanding. The Combined Uncertainty Uc (although it appears only in method 2) assumes uncertainty evaluated at approx 95% coverage probability i.e. at k = Guard Banding at Measurement Uncertainty : ILAC-G8 [6], ISO [7] recommend to take measurement uncertainty into account directly (without any factor), while taking Pass/Fail decision or making compliance statement. ILAC-G8 cl. 2.6 states - In calibration, measurement uncertainty shall always be taken into account when compliance with specifi cation is made. Thus, the Tolerance limits (TL) are set by subtracting the uncertainty from the specifi cation limits (SL). The Guard-band in Fig. 2 is approximately set at uncertainty for the sake of illustration. In ILAC-G8 this fi gure is shown without Guardband & Pass/Fail legends. The diamonds in the fi gure are various measurement results with uncertainty bands above & below. Clause 2.3 of ILAC-G8 recommends the following compliance / non-

4 compliance criteria (which is more or less same for other methods as well): Table 2 : Values of TL for Guard-banding at uncertainty (a) Compliance : If the specifi cation limit is not breached by the measurement result plus the expanded uncertainty with a 95% coverage probability, then compliance with the specifi cation can be stated. In calibration this is often reported as Pass (See Case 1 of Fig. 2) (b) Non-compliance : If the specifi cation limit is exceeded by the measurement result minus the expanded uncertainty with a 95% coverage probability, then noncompliance with the specifi cation can be stated. In calibration this is often reported as Fail (See Case 4 of Fig. 2) Table 3 : Application of uncertainty for statusdecision (c) If the measurement result plus/minus the expanded uncertainty with a 95 % coverage probability overlaps the limit (See Case 2 and 3 of Fig. 2), it is not possible to state compliance or non-compliance. The measurement result and the expanded uncertainty with a 95 % coverage probability should then be reported together with a statement indicating that neither compliance nor non-compliance was demonstrated. In Case 2 of Fig. 2 it is possible to indicate, that the measurement is below the limit, which can be done using a similar statement- It is not possible to state compliance using a 95 % coverage probability for the expanded uncertainty although the measurement result is below the limit The formula to calculate the Test (or Tolerance) limit TL at Guard-band is: Eq. (1) Table 2 shows different values of TL & GB with this method. Table 3 shows application of uncertainty for deciding pass/fail status. Fig. 2 may be referred for decisions regarding the Pass, Fail, Pass 1 & Fail 1 status* (the last column in Table 3). A Guard-band of V has been taken in this table. The decisions under Pass 1 & Fail 1 status are worth noticing & shall be clearly & explicitly reported as explained under (c) above. 4 The method is highly favourable to the consumer & extremely taxing for the producer. Further, it has a large discontinuity at TUR just below 4:1, e.g. even at a TUR of say 3.99:1, TL must be set at 75% of SL; whereas for a TUR of 4.01:1, TL may be set at SL (as it is invoked at a TUR of 4:1). Thus the producer runs comparatively higher risk of False-Rejects for TURs just below 4:1 & enjoys lower risk of False-Rejects for TURs just above 4:1. 2. Guard-banding as per UKAS M3003-M2 As the uncertainty is normally stated at a coverage probability of approximately 95%, statements of compliance will generally be given at that level. But this method (explained in 1) sets quite a wide Guard-band & hence the suppliers run higher risk of False- Reject. A method suggested in UKAS M3003 [8] section M2 allows compliance or non-compliance at 95% confi dence level as well as at other confi dence levels. (It assumes, as per cl. M2.18, that the uncertainty breaches any one of the specifi cation limits and is suffi ciently small so that an insignifi cant portion of the distribution approaches the other limit). The higher risk of False-Reject could be reduced

5 by following UKAS M3003-M2, which suggests setting TL inside the SL by a factor, Ks, equal to 1.64 of U c for 95% coverage probability. For a coverage probability of 99%., k s is equal to 2.32 of U c. Hence the formula to calculate the test limit at 95% coverage probability is: TL = SL - (U c 1.64) Eq. (2) where, U C is Combined uncertainty. Table 4 shows TL & GB values with this method. Table 4: Values of TL for Guard-banding per UKAS M3003-M2: UKAS M3003 in section M3 suggests the following method in such cases, which is akin to RSS evaluation (The correct name rather would be RDS -Root Difference Square). If both the uncertainty U and the specifi cation SL are stated at the same coverage probability, then the acceptance test limits (TL) are established by the square root of the difference of the squared uncertainty (U) from the squared specifi cation (SL). Hence the formula to calculate the test limit is: Eq. (3) It gives False-Accept risk of less than 0.8% for TURs from 4:1 to 1.5:1. However, the False-Reject risk increases from 2% at TUR of 4:1 to 8.3% at TUR of 1.5:1. Table 5 shows different values of TL & GB with this method. This method gives small False-Accept risk and a lower False-Reject risk than earlier method. It is appropriate when the specifi cation limits have been treated as absolute, analogous to a rectangular probability distribution, as suggested by M3003. This may not always be the case. Further, in case if it is not possible to decide compliance/noncompliance at 95% CL, cl. M 2.12 states to reduce the uncertainty, possibly by applying corrections to the result, using more accurate equipment or by taking the mean of a large number of readings. Such an approach, it says, should be taken by agreement with and understanding of the customer. Clause M2.16 of M3003 gives values of Ks from 0.52 to 3.29 corresponding to coverage probabilities from 70% to 99.9%. The formula for k s relates to information about U c, the result of measurement & SL. 3. Guard-banding per UKAS M3003-M3 : Some manufacturers mention specifi cations in their manual at certain confi dence level, typically at 95%. It is stated in GUM that, when a specifi cation is given at a given coverage probability, then a normal distribution can be assumed. Table 5: Values of TL for Guard-banding per M3003-M3 4. RSS Method : It appears in a paper by Deaver David [9] & gives the following equation. Eq. (4) Table 6 shows different values of TL & GB using equation 4. This gives lower False- Accept risk; but higher False-Reject risk than the previous method a). {Also compare this formula & table with that in method 1, Eqn. (1)} 5

6 Table 6: Values of TL for Guard-banding by RSS 5. Guard-banding per NCSL Recommended Practice 10 (RP-10) : The method described in the NCSL- RP10 [10] is given by the following formula: greater than that which would result from using a 4:1 ratio. To accomplish this, the TL is reduced by a fraction M such that False- Accept Risk is equivalent to that of a 4:1 TUR. Thus, TL = SL M The False-Accept Risk is 0.79 % for a TUR of 4:1. With multiplier M as shown in table 8, the False-Accept Risk is maintained less than 0.79 % and the False-Reject Risk is maintained under reasonable limit, provided both uncertainty and specifi cation are given at 95% confi dence level. Table 8 also provides some values of TL & GB for different TURs. For all TURs, the False-Accept risk is restricted to about 0.77%. Table 8: Values of M & TL for Guard-banding per MIL-STD A & ANSI Z540.1 Eq. (5) This equation can be applied for TURs of 4:1 & worse. (For TURs better than 4:1, formula gives TL > SL) As seen from the Table 7, for TUR of 4:1, TL = SL. This method is slightly better than method 1 from producer s risk perspective as it is not discontinuous even just below 4:1 TUR. It gives competitively low False-Accept Risk but still high False-Reject Risk. Table 7 shows different values of TL & GB using this equation. Table 7: Values of TL for Guard-banding per NCSL-RP Guard-Banding Per MIL-STD-45662A & ANSI Z540.1: The old US military standard MIL-STD A and the old ANSI Z540.1 stipulate a TUR of more than 4:1. But when it is not possible to follow this ratio, test limits can beset such that the False-Accept risk is no 6 The method is in the best interest of supplier as False-Accept risk is far less than ANSI Z540.3 requirement. In contrast, the False-Reject risk being high, it is burdensome for the producer. 7. Guard-banding by Managed Risk : The new ANSI Z540.3: 2006, states that the False-Accept risk should be capped to 2% when pass/fail decisions are to be taken. Michael Dobbert of Agilent Technologies in his paper [11] proposed a very innovative method to achieve it, which caps the False-Accept Risk to 2% with limited knowledge of the a priori probability that a device is in-tolerance. With this method, the False-Reject risk is also reduced to a great extent. The test limits (TL) in this method are established by equation (6) TL PFA2% = SL U 95% [1.04 e (0.38ln(TUR)-0.54) ] Eq. (6)

7 Using this method Table 9 shows values of TL & GB for TURs 4:1 & below. (At TURs above 4:1, TLs become more than 100) The above equation can also be written as- TL PFA 2% = SL U M, where the 95% multiplier M is M = 1.04 e (0.38ln(TUR)-0.54) Eq. (7) Table 9 : Values of TL for Guard-banding by Managed Risk Fig. 3 : Comparison of three methods A summary of all methods is given in the Table 11. The SL in all tables is taken as 100 for ease & convenience. The TL is the proportion of SL & be suitably modifi ed as per actual specifi cations used. Table 11: Summary of values of TL for various methods discussed. The values for M are given in Table 10. Table 10: Values of Multiplier M for Managed risk method # TL here is curtailed to 100 for TURs above 4:1 The graphs in Fig. 3, from Michael Dobbert s paper, show the comparison of No- Guard-band, 95% uncertainty Guard-band & Managed risk Guard-band. The advantages of the Managed risk method are obvious from the graphs for both the supplier & the consumer. The method is very inventive as it meets the ANSI Z criteria of False- Accept risk not more than 2% on one hand & balances the false-reject risk on the other. Fig. 4 is a graphical presentation of Table 11. The interpolation is done by drawing smooth curve. More intermediate values of TUR would give smoother curve & interpolation would be more fi ne. Also note that, for TURs better that 4:1, graph for Managed risk is curtailed up-to 100 as their calculated values are more than 100. For NCSL RP-10, MIL-STD-45662A & ANSI Z540.1, TURs better than 4:1 are not applicable. Fig. 4: Tolerance Limit (normalised to 100 %) vs. TUR for various methods 7

8 CONCLUSION: There are various methods being adopted for deciding a Guard-band. Depending upon the Guard-banding technique chosen, both False-Accept & False-Reject risks vary considerably. ILAC-G8, M3003-M2, and RP-10 methods are based on the conditional False-Accept risk while Managed Risk, & M3003-M3 methods are based on the unconditional False-Accept risk. The 95% expanded uncertainty guard band as per ILAC-8, is the widest one & offers lowest False-Accept risk but highest False-Reject Risk. ILAC-G8 allows other than 95% coverage probability for the expanded uncertainty subject to agreement between the laboratory/supplier and the customer. It also states that coverage probabilities for the expanded uncertainty higher than 95 % might be chosen while lower values should be avoided. M3003-M2 comes next to ILAC-G8 method as regards Guard-band & closely follows it throughout TURs from 1.5:1 to 10:1. RSS and RP-10 methods provide closer Guard-bands than M3003-M2 & ILAC-G8, and follow closely till a TUR of 1.5:1 to 4:1 & provide a better False-Reject Risk than 95% expanded uncertainty guard band but still are on higher side compared to UKAS M3003-M3 & Managed risk methods. Managed Risk and UKAS M3003-M3 methods provide comparable results with a controlled False-Accept Risk. The False- Reject risk is signifi cantly lower than ILAC-G8, M3003-M2, RSS and NCSL RP-10 methods. Both can be applied either with specifi cation quoted as absolute limit or when specifi cation are stated with normal distribution. MIL-STD-45662A & ANSI Z give smallest Guard-band & hence pose highest False-Accept risk among Guard-banding methods discussed. MIL-STD-45662A, ANSI Z 540.1, NCSL RP-10 & Managed Risk methods are applicable for TURs from 4:1 & below; while ILAC G-8, UKAS M3003 & RSS methods can be applied up-to 10:1 TUR. All methods are pertinent but shall be used in right context. The method to be followed shall be an informed decision with the customer in the loop. The signifi cance, criticality and reliability of the application (e.g. space, military or industrial) as well as the fi nancial implications should be factored in while adopting any of the methods. The ASME B working group [12] states that the selection of a decision rule is a business decision, and the fl exibility of having a continuum of rules ranging from stringent to relaxed acceptance or rejection is needed in order to satisfy a broad range of industries. I sincerely thank Shri. Gautam Pal, Director, ETDC, Pune for the encouragement & support provided to publish this paper. REFERENCES : [1] ISO17025:2005, General requirements for the competence of testing and calibration laboratories. [2] ANSI/NCSL Z , Requirements for the Calibration of Measuring and Test Equipment, National Conference of Standard Laboratories, [3] Guide to the Expression of Uncertainty in Measurement (GUM), BIPM, IEC, IFCC, ISO, IUPAC, IUPAP, OIML - International Organization for Standardization, [4] Military Standard 45662A-Calibration System Requirements, United States of America, Department of Defence, [5] ANSI/NCSL Z , Calibration Laboratories and Measuring and Test Equipment - General Requirements, National Conference of Standard Laboratories International, [6] ILAC-G8:03/2009: Guidelines on the Reporting of Compliance with Specifi cation. 8

9 [7] ISO :1998, Geometrical Product Specifi cations (GPS) -- Inspection by measurement of workpieces and measuring equipment -- Part 1: Decision rules for proving conformance or nonconformance with specifi cations, International Organization for Standardization, [8] UKAS M3003:2007, The Expression of Uncertainty and Confi dence in Measurement, United Kingdom Accreditation Service, 2007, pp [9] Deaver, David K., Guard-banding and the World of ISO Guide 25. Is There Only One Way? 1998 NCSL International Workshop & Symposium. [11] Dobbert, Michael, A Guard-Band Strategy for Managing False-Accept Risk, 2008 NCSL International Workshop & Symposium. [12] ASME B Guidelines for Decision Rules: considering Measurement Uncertainty in Determining Conformance with Specifi cations. [13] Marcello Lucano, Differences in Guard-banding Strategies-A Beginner s Guide Agilent Technologies Italia S.p.A. [14] Deaver, David K., How to Maintain Your Confi dence (In a World of Declining Test Uncertainty Ratios), 1993 NCSL International Workshop & Symposium [10] NCSL RP-10: Establishment and Operation of an Electrical Utility Metrology Laboratory, National Conference of Standard Laboratories,