equivalence Intelligent Statistical Methods for Safer and More Robust Qualifications Wayne J. Levin M.A.Sc. P.Eng President: Predictum Inc. Disclaimer: Predictum Inc. will not be liable for any loss or damage which you may suffer as a result of or connected with the download, viewing or application of methods and advice contained in this file. PREDICTUM INC. WWW.PREDICTUM.COM 2010-2011
items Why Qualifications Fail: Engineers Top 10 List How many do I really need to measure? How do I know they are the same? Statistical Methods Assuring Equivalence thoughts and recommendations
Deming s production viewed as a system Suppliers of materials & equipment Design & Redesign Consumer Research A B Production & test of materials Production, assembly, inspection Distribution Consumers C D Tests of processes, machines, methods, costs W. Edwards Deming, Out of the Crisis, p.2
top 10 risks in qualification Disclaimer: Predictum Inc. will not be liable for any loss or damage which you may suffer as a result of or connected with the download, viewing or application of methods and advice contained in this file. PREDICTUM INC. WWW.PREDICTUM.COM 2010-2011
Full Risk is not analyzed A full risk analysis is not done beforehand; purpose(s) of the process, inputs, outputs, measures. The qualification plan is not based on this full risk analysis. A Secondary purpose or effect is not accounted for While the primary purpose of the process is accounted for, possible secondary purposes are not discussed or known and thus potentially not examined and checked. Insufficient Sample Size The sample size used cannot detect shifts with reasonable probability. The probability of missing a shift is not examined up front and thus relative risk is not known.
Unexamined Sources of Variation The potential sources of variation were not examined beforehand. Sample and run design does not take into account relevant potential sources of variation. Interactions not tested; Unwanted bias not controlled Interactions between possible factors are not known and not tested. The factor being changed and qualified may interact with other factors that are not being tested.
Unwanted bias not controlled or Covariates unaccounted Runs are not randomized but made with intended or unintended patterns; shift to shift, tool to tool, Device to Device, etc.. These patterns are not accounted for in the analysis. Variables that cannot be controlled but may be significant are not measured and included in the analysis. Qualification testing fails to include variation Testing for equivalence is done only on the process average but fails to include potential changes in process variation. This is especially dangerous for low Cpk processes.
Testing Equivalence using the wrong control The control sample includes sources of variation that are special cause related or not included in the qualification sample. This inflates the variation used in testing and can hide potentially significant shifts. Shift within Specification Wide specification limits allow for a shift. This shift may bring the process into an area never run in before, but still within specification.
Future results are not simulated Given the shift due to the change, the expectation of future performance and Cpk is not simulated up front. No follow-up checks and/or no monitors to catch drift Critical parameters are not checked post change. No follow-up is done to check for stability of the change.
ACT PLAN STUDY DO
ACT PLAN STUDY DO ACT PLAN STUDY DO
Decide whether to put the tool/material into production PLAN Full Risk Analysis Variance Analysis Power Analysis Equivalence Value ACT ACT PLAN STUDY DO STUDY DO Oneway ANOVA Equivalence Unequal Variance Test Normal Quantile Plot QQ-Plot Simulation / Cpk Analysis Collect Data
plan:sampling HOW MANY? FROM WHAT LOCATION? Disclaimer: Predictum Inc. will not be liable for any loss or damage which you may suffer as a result of or connected with the download, viewing or application of methods and advice contained in this file. PREDICTUM INC. WWW.PREDICTUM.COM 2010-2011
PLAN Full Risk Analysis Variance Analysis Power Analysis Equivalence Value ACT ACT PLAN STUDY DO STUDY DO
plan:sampling representative: capture the dominant sources of process variation look for patterns in your variation sufficient: sample enough data to be sure to see a difference if it is there
edge effect Though these wafers went through one step, there are, apparently, two process conditions in effect: outer dies versus inner dies
edge effect A 3D view of the same wafers
Deming s production viewed as a system Suppliers of materials & equipment Design & Redesign Consumer Research A B C Production & test of materials Production, assembly, inspection Distribution Consumers OUTER INNER D Tests of processes, machines, methods, costs W. Edwards Deming, Out of the Crisis, p.2
variance components within wafer wafer-to-wafer lot-to-lot
variance components Lot-to-lot variation is the highest component: 2/3 wafer-to-wafer, within lot is 28% Residual = within wafer (site-to-site) variation
Moose Jaw 2708 KM Toronto
power Power is the ability to detect a difference if it is there. Usually want power to be above 0.80.
microscope: the more powerful, the more insights that can be seen
power Power is affected by: sample size: the more data, the higher power alpha: the higher alpha risk, the higher power standard deviation: the more variability, the less power delta: bigger differences are easier to see (higher power)
power:sample size
power:alpha
power:variability
power:delta
plan:equivalence How much of a difference is considered equivalent?
PLAN ACT ACT PLAN STUDY DO STUDY DO
study Disclaimer: Predictum Inc. will not be liable for any loss or damage which you may suffer as a result of or connected with the download, viewing or application of methods and advice contained in this file. PREDICTUM INC. WWW.PREDICTUM.COM 2010-2011
PLAN ACT ACT PLAN STUDY DO STUDY DO Oneway ANOVA Equivalence Unequal Variance Test Normal Quantile Plot QQ-Plot Simulation / Cpk Analysis
hypothesis testing Disclaimer: Predictum Inc. will not be liable for any loss or damage which you may suffer as a result of or connected with the download, viewing or application of methods and advice contained in this file. PREDICTUM INC. WWW.PREDICTUM.COM 2010-2011
Concerning Hypothesis Testing: Small wonder that students have trouble. They may be trying to think. W. Edwards Deming On Probability as a Basis for Action The American Statistician, Vol.29, No. 4, 1975, pp 146-152
plan:sampling Step 1: Know your Null Hypothesis H 0 for equivalence: the pre & post processes average and variances are the same Step 2: Determine alpha alpha is inversely proportional to risk Step 3: Collect your data and conduct analysis Step 4: Compare p-value to alpha p > alpha: Fail to reject H 0 (accept H 0 ) p < alpha: reject H 0 Reject H0 Fail to Reject H0
errors Type I False Signal make a change that does not result in improvement controlled by alpha: occurs alpha% of the time Type II Missed opportunity fail to make a change that would result in improvement controlled by beta (determined by alpha, sample size, variability and the difference you wish to detect)
example 1 Step 1: H 0 for equivalence: the pre & post processes averages are the same Step 2: alpha = 0.05 Step 3: Data complete - analysis at right Step 4: p > alpha: Fail to reject H 0 (accept H 0 )
example 2 Step 1: H 0 for equivalence: the pre & post processes averages are the same Step 2: alpha = 0.05 Step 3: Data complete - analysis at right Step 4: p < alpha: reject H 0 (accept H 0 )
equivalence testing Disclaimer: Predictum Inc. will not be liable for any loss or damage which you may suffer as a result of or connected with the download, viewing or application of methods and advice contained in this file. PREDICTUM INC. WWW.PREDICTUM.COM 2010-2011
equivalence testing Same difference for both 26 observations p-value > 0.2 Fail to reject H0 100 observations p-value < 0.05 Reject H0 Any difference, no matter how small, can be found to be significantly different, if the sample size is big enough
equivalence establish a minimum difference is to be considered practically zero if the actual difference is = to a this difference, then declare PRE POST if the actual difference is closer to zero, then declare PRE= POST
different the actual difference (1.8) is not significantly different than the difference considered to be practically zero (2) a high p-value means different
equivalent the actual difference (0.22) is significantly different than the difference considered practically zero (2) Note the low p- values indicated that actual difference is significantly different than the practical difference
evaluating equivalence location, variance & shape Disclaimer: Predictum Inc. will not be liable for any loss or damage which you may suffer as a result of or connected with the download, viewing or application of methods and advice contained in this file. PREDICTUM INC. WWW.PREDICTUM.COM 2010-2011
all equivalent normal quantile plot points on top of each other
H0: the variances are equal alpha = 0.05 p > alpha fail to reject H0
different distributions Variances are different Normal quantile plot slopes are different
H0: the variances are equal alpha = 0.05 p > alpha reject H0
the shape of a distribution is not easily determined with a control chart
location & variation are different Normal quantile plot has different slopes points follow their respective lines indicating that they are normally distributed
H0: the variances are equal alpha = 0.05 p > alpha reject H0
same location different variation normal quantile plots have different slopes
Note that the normal quantile plot intersect each other at the center p-values are low, reject H0 that the variances are the same
note the average is the same across both PRE and POST Control Limits show change in variation
location different, variance the same Normal quantile plot slopes are the same Normal quantile plot slopes are separated
Control limits are the same width processes are centered differently
study:q-q Plot Quantile- Quantile plot shows equivalence among location variation shape
equivalent The points follow the diagonal line Equivalent average variability
Different distributions off-diagonal curvature
same distribution, variance different location points parallel to the diagonal
different variability & location no curvature - same distribution different slope - different variability points intersect red line offcenter - different location
different variability & location no curvature - same distribution different slope - different variability points intersect red line at center - same location
Decide whether to put the tool/material into production PLAN ACT ACT STUDY PLAN DO STUDY DO
thoughts and recommendations THE NEW NORMAL Disclaimer: Predictum Inc. will not be liable for any loss or damage which you may suffer as a result of or connected with the download, viewing or application of methods and advice contained in this file. PREDICTUM INC. WWW.PREDICTUM.COM 2010-2011
sample size:a big challenge more data is required to determine the shape of a distribution
reduce variation! High variability, low power. Each additional sample provides very little power Low variability, high power. Each additional sample provides a lot more power
cooperate with your suppliers! Suppliers of materials & equipment Design & Redesign Consumer Research A B Production & test of materials Production, assembly, inspection Distribution Consumers C D Tests of processes, machines, methods, costs W. Edwards Deming, Out of the Crisis, p.2
3. Cease dependence on inspection to achieve quality. The 14 points W. Edwards Deming Out of the Crisis, MIT 1982 Quals are inspections. Innovate to reduce the need to perform quals Build systems that facilitate diligence and maintain stability (not easy to do - but imagine the benefits to yield, quality & productivity)
smed single minute exchange of die changed the system
1. Create constancy of purpose for improvement of product and service. The 14 points W. Edwards Deming Out of the Crisis, MIT 1982
Control In control process maximum productivity given the current system change the system to achieve higher levels of productivity
items Why Qualifications Fail: Engineers Top 10 List How many do I really need to measure? How do I know they are the same? Statistical Methods Assuring Equivalence thoughts and recommendations
Questions?
how can we help? Wayne J. Levin M.A.Sc. P.Eng levin@predictum.com (416) 398-8900