Intelligent Statistical Methods for Safer and More Robust Qualifications

Transcription

1 equivalence Intelligent Statistical Methods for Safer and More Robust Qualifications Wayne J. Levin M.A.Sc. P.Eng President: Predictum Inc. Disclaimer: Predictum Inc. will not be liable for any loss or damage which you may suffer as a result of or connected with the download, viewing or application of methods and advice contained in this file. PREDICTUM INC

2 items Why Qualifications Fail: Engineers Top 10 List How many do I really need to measure? How do I know they are the same? Statistical Methods Assuring Equivalence thoughts and recommendations

3 Deming s production viewed as a system Suppliers of materials & equipment Design & Redesign Consumer Research A B Production & test of materials Production, assembly, inspection Distribution Consumers C D Tests of processes, machines, methods, costs W. Edwards Deming, Out of the Crisis, p.2

4 top 10 risks in qualification Disclaimer: Predictum Inc. will not be liable for any loss or damage which you may suffer as a result of or connected with the download, viewing or application of methods and advice contained in this file. PREDICTUM INC

5 Full Risk is not analyzed A full risk analysis is not done beforehand; purpose(s) of the process, inputs, outputs, measures. The qualification plan is not based on this full risk analysis. A Secondary purpose or effect is not accounted for While the primary purpose of the process is accounted for, possible secondary purposes are not discussed or known and thus potentially not examined and checked. Insufficient Sample Size The sample size used cannot detect shifts with reasonable probability. The probability of missing a shift is not examined up front and thus relative risk is not known.

6 Unexamined Sources of Variation The potential sources of variation were not examined beforehand. Sample and run design does not take into account relevant potential sources of variation. Interactions not tested; Unwanted bias not controlled Interactions between possible factors are not known and not tested. The factor being changed and qualified may interact with other factors that are not being tested.

7 Unwanted bias not controlled or Covariates unaccounted Runs are not randomized but made with intended or unintended patterns; shift to shift, tool to tool, Device to Device, etc.. These patterns are not accounted for in the analysis. Variables that cannot be controlled but may be significant are not measured and included in the analysis. Qualification testing fails to include variation Testing for equivalence is done only on the process average but fails to include potential changes in process variation. This is especially dangerous for low Cpk processes.

8 Testing Equivalence using the wrong control The control sample includes sources of variation that are special cause related or not included in the qualification sample. This inflates the variation used in testing and can hide potentially significant shifts. Shift within Specification Wide specification limits allow for a shift. This shift may bring the process into an area never run in before, but still within specification.

9 Future results are not simulated Given the shift due to the change, the expectation of future performance and Cpk is not simulated up front. No follow-up checks and/or no monitors to catch drift Critical parameters are not checked post change. No follow-up is done to check for stability of the change.

10 ACT PLAN STUDY DO

11 ACT PLAN STUDY DO ACT PLAN STUDY DO

12 Decide whether to put the tool/material into production PLAN Full Risk Analysis Variance Analysis Power Analysis Equivalence Value ACT ACT PLAN STUDY DO STUDY DO Oneway ANOVA Equivalence Unequal Variance Test Normal Quantile Plot QQ-Plot Simulation / Cpk Analysis Collect Data

13 plan:sampling HOW MANY? FROM WHAT LOCATION? Disclaimer: Predictum Inc. will not be liable for any loss or damage which you may suffer as a result of or connected with the download, viewing or application of methods and advice contained in this file. PREDICTUM INC

14 PLAN Full Risk Analysis Variance Analysis Power Analysis Equivalence Value ACT ACT PLAN STUDY DO STUDY DO

15 plan:sampling representative: capture the dominant sources of process variation look for patterns in your variation sufficient: sample enough data to be sure to see a difference if it is there

16 edge effect Though these wafers went through one step, there are, apparently, two process conditions in effect: outer dies versus inner dies

17 edge effect A 3D view of the same wafers

18 Deming s production viewed as a system Suppliers of materials & equipment Design & Redesign Consumer Research A B C Production & test of materials Production, assembly, inspection Distribution Consumers OUTER INNER D Tests of processes, machines, methods, costs W. Edwards Deming, Out of the Crisis, p.2

19 variance components within wafer wafer-to-wafer lot-to-lot

20 variance components Lot-to-lot variation is the highest component: 2/3 wafer-to-wafer, within lot is 28% Residual = within wafer (site-to-site) variation

21 Moose Jaw 2708 KM Toronto

22 power Power is the ability to detect a difference if it is there. Usually want power to be above 0.80.

23 microscope: the more powerful, the more insights that can be seen

24 power Power is affected by: sample size: the more data, the higher power alpha: the higher alpha risk, the higher power standard deviation: the more variability, the less power delta: bigger differences are easier to see (higher power)

25 power:sample size

26 power:alpha

27 power:variability

28 power:delta

29 plan:equivalence How much of a difference is considered equivalent?

30 PLAN ACT ACT PLAN STUDY DO STUDY DO

31 study Disclaimer: Predictum Inc. will not be liable for any loss or damage which you may suffer as a result of or connected with the download, viewing or application of methods and advice contained in this file. PREDICTUM INC

32 PLAN ACT ACT PLAN STUDY DO STUDY DO Oneway ANOVA Equivalence Unequal Variance Test Normal Quantile Plot QQ-Plot Simulation / Cpk Analysis

33 hypothesis testing Disclaimer: Predictum Inc. will not be liable for any loss or damage which you may suffer as a result of or connected with the download, viewing or application of methods and advice contained in this file. PREDICTUM INC

34 Concerning Hypothesis Testing: Small wonder that students have trouble. They may be trying to think. W. Edwards Deming On Probability as a Basis for Action The American Statistician, Vol.29, No. 4, 1975, pp

35 plan:sampling Step 1: Know your Null Hypothesis H 0 for equivalence: the pre & post processes average and variances are the same Step 2: Determine alpha alpha is inversely proportional to risk Step 3: Collect your data and conduct analysis Step 4: Compare p-value to alpha p > alpha: Fail to reject H 0 (accept H 0 ) p < alpha: reject H 0 Reject H0 Fail to Reject H0

36 errors Type I False Signal make a change that does not result in improvement controlled by alpha: occurs alpha% of the time Type II Missed opportunity fail to make a change that would result in improvement controlled by beta (determined by alpha, sample size, variability and the difference you wish to detect)

37 example 1 Step 1: H 0 for equivalence: the pre & post processes averages are the same Step 2: alpha = 0.05 Step 3: Data complete - analysis at right Step 4: p > alpha: Fail to reject H 0 (accept H 0 )

38 example 2 Step 1: H 0 for equivalence: the pre & post processes averages are the same Step 2: alpha = 0.05 Step 3: Data complete - analysis at right Step 4: p < alpha: reject H 0 (accept H 0 )

39 equivalence testing Disclaimer: Predictum Inc. will not be liable for any loss or damage which you may suffer as a result of or connected with the download, viewing or application of methods and advice contained in this file. PREDICTUM INC

40 equivalence testing Same difference for both 26 observations p-value > 0.2 Fail to reject H0 100 observations p-value < 0.05 Reject H0 Any difference, no matter how small, can be found to be significantly different, if the sample size is big enough

41 equivalence establish a minimum difference is to be considered practically zero if the actual difference is = to a this difference, then declare PRE POST if the actual difference is closer to zero, then declare PRE= POST

42 different the actual difference (1.8) is not significantly different than the difference considered to be practically zero (2) a high p-value means different

43 equivalent the actual difference (0.22) is significantly different than the difference considered practically zero (2) Note the low p- values indicated that actual difference is significantly different than the practical difference

44 evaluating equivalence location, variance & shape Disclaimer: Predictum Inc. will not be liable for any loss or damage which you may suffer as a result of or connected with the download, viewing or application of methods and advice contained in this file. PREDICTUM INC

45 all equivalent normal quantile plot points on top of each other

46 H0: the variances are equal alpha = 0.05 p > alpha fail to reject H0

47

48 different distributions Variances are different Normal quantile plot slopes are different

49 H0: the variances are equal alpha = 0.05 p > alpha reject H0

50 the shape of a distribution is not easily determined with a control chart

51 location & variation are different Normal quantile plot has different slopes points follow their respective lines indicating that they are normally distributed

52 H0: the variances are equal alpha = 0.05 p > alpha reject H0

53

54 same location different variation normal quantile plots have different slopes

55 Note that the normal quantile plot intersect each other at the center p-values are low, reject H0 that the variances are the same

56 note the average is the same across both PRE and POST Control Limits show change in variation

57 location different, variance the same Normal quantile plot slopes are the same Normal quantile plot slopes are separated

58

59 Control limits are the same width processes are centered differently

60 study:q-q Plot Quantile- Quantile plot shows equivalence among location variation shape

61 equivalent The points follow the diagonal line Equivalent average variability

62 Different distributions off-diagonal curvature

63 same distribution, variance different location points parallel to the diagonal

64 different variability & location no curvature - same distribution different slope - different variability points intersect red line offcenter - different location

65 different variability & location no curvature - same distribution different slope - different variability points intersect red line at center - same location

66 Decide whether to put the tool/material into production PLAN ACT ACT STUDY PLAN DO STUDY DO

67 thoughts and recommendations THE NEW NORMAL Disclaimer: Predictum Inc. will not be liable for any loss or damage which you may suffer as a result of or connected with the download, viewing or application of methods and advice contained in this file. PREDICTUM INC

68 sample size:a big challenge more data is required to determine the shape of a distribution

69 reduce variation! High variability, low power. Each additional sample provides very little power Low variability, high power. Each additional sample provides a lot more power

70 cooperate with your suppliers! Suppliers of materials & equipment Design & Redesign Consumer Research A B Production & test of materials Production, assembly, inspection Distribution Consumers C D Tests of processes, machines, methods, costs W. Edwards Deming, Out of the Crisis, p.2

71 3. Cease dependence on inspection to achieve quality. The 14 points W. Edwards Deming Out of the Crisis, MIT 1982 Quals are inspections. Innovate to reduce the need to perform quals Build systems that facilitate diligence and maintain stability (not easy to do - but imagine the benefits to yield, quality & productivity)

72 smed single minute exchange of die changed the system

73 1. Create constancy of purpose for improvement of product and service. The 14 points W. Edwards Deming Out of the Crisis, MIT 1982

74 Control In control process maximum productivity given the current system change the system to achieve higher levels of productivity

75 items Why Qualifications Fail: Engineers Top 10 List How many do I really need to measure? How do I know they are the same? Statistical Methods Assuring Equivalence thoughts and recommendations

76 Questions?

77 how can we help? Wayne J. Levin M.A.Sc. P.Eng (416)