Measurement and Sigma Metrics

Size: px

Start display at page:

Download "Measurement and Sigma Metrics"

Elijah Ferguson
6 years ago
Views:

2 Measurement is the first step that leads to control and eventually to improvement. If you can t measure something, you can t understand it. If you can t understand it, you can t control it. If you can t control it, you can t improve it RCPA Quality Assurance Programs Pty Ltd. All rights reserved.

4 EQA driving Improvement 90% of the EQA problems were resolved after the first round of EQA and 99% were resolved by the third round, suggesting that the laboratories had successfully corrected mistakes identified by EQA performance. Hoeltge FA, Phillips MG, Styer PE, Mockridge P (2005) Detection and correction of systematic laboratory problems by analysis of clustered proficiency testing failures. Arch Pathol Lab Med.129(2): Poor PT/EQA performance because of interpretation errors has decreased between 1997 and This may be derived from continued participation in PT/EQA. Ramsden SC, Deans Z, Robinson DO, et al (2006) Monitoring standards for molecular genetic testing in the United Kingdom, The Netherlands and Ireland. Genetic Test 10(3):

7 Roadmap to selecting the optimal Westgard QC rule for a test Collect the general test characteristics CVa, bias and TEa CVa (%): calculate the total analytical variation coefficient (intra and inter analytical) from a period long enough to yield a representative impression of the performance of the test (it is recommended to discard outliers). This information can be derived from results of your internal QC program. Bias (%): calculate the systematic difference between the mean of your test results compared to the mean of the results acquired with a reference method or of a group mean. This information can be collected from results of an external QC program and should also cover a representative period. In fact, if comparison with a reference method is not possible, consider using state-of-art or consensus values.

8 Roadmap to selecting the optimal Westgard QC rule for a test 1. TEa (%): read the value of TEa is defined as desirable specification for allowable error and referred to as desirable TE or TEd by Ricos et al.). 2. Calculate the Sigma value of the test using the formula: Sigma as (TEa bias)/cva. 3. Select the optimal Westgard QC (multi)rule. As the Sigma value of the Westgard rules decreases, stringency increases as does the chance of missing an error (p error detection decreases). Therefore, the best Westgard rule is the one with a Sigma value closest to, but smaller than, the Sigma value of the test.

9 Capability Classes Cps less than 3 : Incapable Cps between 3 and 4: Barely capable Cps between 4 and 6: Capable Cps greater than 6: Highly capable (World class) Defects per million 3 sigma 2700/ sigma 63/ sigma.57/ sigma.002/ RCPA Quality Assurance Programs Pty Ltd. All rights reserved.

13 Allowable Limits of Performance based on Biological Variability Monitoring Diagnostic

22 What else could we do with EQA data?

23 Identifying problem assays with EQA using Sigma metrics Achievable Standard: Assay Capability of 20th percentile QAP laboratory Cps<4 4<Cps<6 Cps>6 Na, K, Cl, CO 2, Ca, Ck, Tbil, Osml, hcg, drugs, CKMB, Ferr*, Fruc* Urea, Creat, Mg, PO 4, Alb, Alp, Alt, Ast, Urate, Lactate, Trig, HDL, Chollipids, Cbil-neonate* Glucose, Cbil, Prot, GGT, Amyl, Lip, LD, Chol, Fe, TIBC, Trf, Li <20% of all labs have Cps % of all labs have Cps % of all labs have Cps 4

24 Peer Group Capability 0-4 Peer Group Capability 4-6 Peer Group Capability > 6 Our Laboratory Better than peer- Better than peer- As good as peer- Capability group cell group cell group cell >6 Our Laboratory Better than peer- As good as peer- Worse than peer- Capability group cell group cell group cell 4-6 Our Laboratory As good as peer- Worse than peer- Worse than peer- Capability group cell group cell group cell 0-4

26 Measurement Comparison to standard Sigma compared to Allowable Limit or Total Error What is Allowable limit with pre-analytical error? Haemolysis? ID mixup? 2012 RCPA Quality Assurance Programs Pty Ltd. All rights reserved.

27 Measure Frequency of Error Compare frequency and compare against peers Prioritise problems and identify most significant Significant is defined as: Hard to Identify/Fix Most impact on patient outcome FMEA 2012 RCPA Quality Assurance Programs Pty Ltd. All rights reserved.

34 Fig. 1. EQA/PT program B, percentage of chemistry assays expected to achieve over 5 Sigma performance, from 27 chemistry tests. Sten A. Westgard Utilizing global data to estimate analytical performance on the Sigma scale: A global comparative analysis of methods, instruments, and manufacturers through external quality assurance and proficiency testing programs Clinical Biochemistry, 2016, Available online 4 March RCPA Quality Assurance Programs Pty Ltd. All rights reserved.

35 Metric M= f i *m i ; f i * could depend on the importance of the measurand therefore become f i *m i ; w i is the weighting given for the measurand n i. Hence M = f i *m i / n i w i. The metric could further be modified to include a cost of test, for example M = (f i *m i + g i *c i ) / n i w i, where g i is a weight per test i (which may be the same as f i ) and c i is the cost of test i.

36 Summary Measurement improves performance Need to have a comparator Six sigma assists with application of QC rules Six sigma concepts should be applied with some caution particularly in relation to ALE

42 We can represent this as: Xr Xc = AG = bias + SEcont*SD *SD, where AG is analytical goal, SEcont*S represents the size of error detected by the QC algorithm, 1.65 gives a 5% chance of results falling outside the acceptable range and SD is the measured imprecision. Next we consider the critical shift a QC algorithm must detect at 90% Ped; SEcrit = [(AG bias)/imprecision] 1.65 If we express all the components in SD units then, SEcrit = Cpa 1.65; and if bias is AG/8, SEcrit = Cpa*7/ RCPA Quality Assurance Programs Pty Ltd. All rights reserved.

43 Assay Capability, Cpa = AG/CVa Assay Capability to maintain 90% Error Detection versus SEdrift % 5% 10% 15% 20% 25% 30% 35% 40% 45% SEdrift (bias as fraction of Analytcal Goal (AG) Cpa if using MR1 Cpa if using MR RCPA Quality Assurance Programs Pty Ltd. All rights reserved.

44 We have shown that two points of Westgard approach need to be corrected. The first is that in the presence of bias, SM CV and SM σ are not identical and we cannot use CV instead of σ. The second is that the SM value obtained from SM CV and SM σ is always lower than the real SM σ. To calculate the SM of a process, we should use the z transformation instead of SM CV and SM σ RCPA Quality Assurance Programs Pty Ltd. All rights reserved.

45 In the N distribution, the data around l are symmetrical and the areas under the curve of both sides of l are equal (a). In the presence of a bias, the curve moves to the side of the bias, but the tolerance limits do not move. Consequently, the areas under the curve on the right and left sides around l are not equal (b). In case c, although the bias is larger than TEa, i.e., l is shifted beyond UL, the SM value of the process remains [0, since a (small) fraction of the area lies within the tolerance limits (the area between LL and UL) 2012 RCPA Quality Assurance Programs Pty Ltd. All rights reserved.

On-Site Routine/STAT Laboratory Tests. This policy provides information regarding approved procedures performed at each site

On-Site Routine/STAT Laboratory Tests. This policy provides information regarding approved procedures performed at each site Purpose: Policy: This policy provides information regarding approved procedures performed at each site This policy provides a list of laboratory tests performed on-site in each AHS Central Zone Northeast