Proposals by the Standing Technical Group on Round Robin (i) Outsourcing the Statistical Evaluation and (ii) Using Robust Statistics

Transcription

1 INTERNATIONAL ASSOCIATION OF OF ASSAY ASSAY OFFICES IAAO 45 Proposals by the Standing Technical Group on Round Robin (i) Outsourcing the Statistical Evaluation and (ii) Using Robust Statistics Note by the Convention Secretariat 1. At its last meeting in Geneva on 3 February 2010, the Standing Technical Group (STG) discussed two proposals: the first is to outsource the statistical evaluation of RR to a professional statistician; and the second is to use robust statistics for the statistical evaluation of RR. 2. Regarding the outsourcing of the statistical evaluation, the STG recommends to the IAAO to appoint Mr. Philippe Caillaud as the statistician in charge of assessing RR 21, 22 & 23. Mr. Caillaud has already made the evaluation of the previous RR (19 & 20). His CV is attached at Annex 4. Mr. Caillaud has also submitted two offers, which are attached (see Annexes 2 and 3). The main difference between the two offers is that in the first offer, the statistical evaluation of IAAO and Convention results is done separately (two distinct sub-reports) while in the second offer, the statistical evaluation of IAAO and Convention results is combined (to a certain extent). 3. For more information on the STG s proposals, see attached extract of Summary Record at Annex 1. The STG has also submitted to the IAAO a proposal to revise the RR Guidelines. * * * * * * *

2 Annex 1 Extract of Summary Record of STG s meeting (PMC/W 1/2010) Round Robin Convention s Test Results 44. The STG reviewed the test results and the report prepared by Mr. Philippe Caillaud on behalf of the Birmingham Assay Office (PMC/INF 33/2009 (Rev. 1)). Ms. Isabel Morais (Portuguese Assay Office) said that future compilations should contain only one list of participants (rather than one list per alloy and method). 45. Mr. Thomas Brodmann (Swiss Assay Office) said that considering the short deadline, given to Mr. Caillaud, the compilation was excellent. The Group agreed that Mr. Caillaud had done a tremendous job and should be used again during future Round Robins. Using the same statistician would provide a greater consistency in RR results. Moreover, the fact that Mr. Caillaud was not with an Assay Office also guaranteed third-party independence. Mr. Caillaud should be appointed by the IAAO at the next meeting based on the STG s recommendation. The RR Guidelines would have to be amended accordingly. It was so agreed. Robust statistics 46. The STG discussed a proposal, made by Mr. Caillaud, to use robust statistics based on ISO for future RR (PMC/W 13/2009). According to Mr. Caillaud, Part 5 of ISO 5725 was more appropriate for RR and had the advantage of calculating the average and the standard deviation on the basis of all data without removing outliers. This was important, since participants did not all use the same method. Under Part 2 of ISO 5725, outliers were identified through two successive tests, Cochran and Grubb. 47. The Group agreed with the proposal, which was considered more appropriate for RR, and suggested amending the RR Guidelines accordingly. Ms. Isabel Morais (Portuguese Assay Office) expressed some reservations: the change from Part 2 to Part 5 of ISO 5725 would not reduce the number of z-scores > 2. Moreover, it might be difficult to apply Part 5 if there were too many bad results or poor quality data.

3 Annex 2 Offer No 1 Philippe CAILLAUD Blanquefort, 12 March rue des Cèdres Blanquefort philippe.caillaud@wanadoo.fr Hallmarking Convention c/o PIC/S 14, rue du Roveray 1207 Geneva Switzerland «Statistical studies on the compilation of Round Robins for precious metal alloys (test program 2010)» Description of services to be rendered by 1 July 2010: 1 ) 12 statistical studies to determine the performance of individual laboratories for specific tests or measurements with round robins tests. Due to the fact that there are three samples (gold, silver and platinum), two organisations (IAAO and Convention) and two testing methods (XRF and chemical testing), there are 12 compilations (= 3 2 2). The statistical analysis for each compilation contains: a ) Results All cell values for mean and standard deviation are calculated. b ) Incoherent results c ) Numeric statistics Two important parameters are calculated: µ = robust mean calculated with the robust algorithm A from ISO or ISO from all quantified results. σ Repro = reproducibility standard deviation of all lab means, calculated with the robust algorithm A from ISO or ISO from all quantified results. d ) Interpretation The Z-score values are calculated and the table accordingly sorted. All results are summarised in a chart (laboratory s mean ± standards deviation and the resulting Z- score). The reference value for the XRF sample is the robust mean of the chemical analysis. 2 ) Two distinct performance reports on each round: one for the IAAO and one for the Convention. Part A: statistical studies for the IAAO. Part B: statistical studies for the HMC (convention). Each statistical report contains: Results and statistical summary of all laboratories Graphics (Z-scores & Histogram). Incoherent results. Remarks. Participants The complete Excel spreadsheets containing all data should also be sent with the reports. Offer 1: Financial Cost Statistical study and drafting of report (in English) For one compilation of results 150 Euros Number of compilations 12 Total amount 1800 Euros

4 Annex 3 Offer No 2 Philippe CAILLAUD Blanquefort, 12 March rue des Cèdres Blanquefort philippe.caillaud@wanadoo.fr Hallmarking Convention c/o PIC/S 14, rue du Roveray 1207 Geneva Switzerland «Statistical studies on the compilation of Round Robins for precious metal alloys (test program 2010)» Description of services to be rendered by 1 July 2010: 1 ) 6 statistical studies to determine the performance of individual laboratories for specific tests or measurements with round robins tests for two organisations (IAAO and Convention). For each statistical studies, there are three samples (gold, silver and platinum) and two testing methods (XRF and chemical testing), there are 6 compilations (= 3 2). The statistical analysis for each organisation and compilation contains: a ) Results All cell values for mean and standard deviation are calculated. b ) Incoherent results c ) Numeric statistics Two important parameters are calculated: µ = robust mean calculated with the robust algorithm A from ISO or ISO from all quantified results. σ Repro = reproducibility standard deviation of all lab means, calculated with the robust algorithm A from ISO or ISO from all quantified results. d ) Interpretation The Z-score values are calculated and the table accordingly sorted. All results are summarised in a chart (laboratory s mean ± standards deviation and the resulting Z- score). The reference value for the XRF sample is the robust mean of the chemical analysis. 2 ) Two distinct performance reports on each round: one for the IAAO and one for the Convention. Each statistical report contains: Results and statistical summary of all laboratories Graphics (Z-scores & Histogram). Incoherent results. Remarks. Participants The complete Excel spreadsheets containing all data should also be sent with the reports. Offer 2: Financial Cost Statistical study and drafting of report (in English) For one compilation of results 300 Euros Number of compilations 6 Total amount 1800 Euros

5 Annex 4 Curriculum Vitae 1 Address & Background Philippe CAILLAUD 1 rue des Cèdres Blanquefort France Tel: Mail: philippe.caillaud@wanadoo.fr Doctor in statistics from the University of Pau (1993) Independent, registered statistician (since 1995) 2 References 2.1 Expert AFNOR expert in normalisation («Association française de Normalisation» (AFNOR) is the French national organisation for standardisation) Chair of AFNOR sub-committee on T90Q (validation of a quantitative method, uncertainty of measurement and control chart). T : protocol for the initial method performance assessment in a laboratory in water. V03-110: protocol of characterization for the validation of a quantitative method of analysis by construction of an accuracy profile in analysis of agrifoodstuffs. XPT : protocol for the estimation of the uncertainty associated to an analysis result for physico-chemical analysis method XPT : protocol for the evaluation of the uncertainty of measurement associated with a microbiological analysis result in water. Formation about statistics, uncertainty of measurement, validation of a quantitative method. OIV expert in normalisation (OIV = International Organisation of Wine) Protocol for the validation of a quantitative method and uncertainty associated to an analysis result in wine. 2.2 Statistician for Round Robin Test Precious metal alloys RR with IAAO (International Association of Assay Offices). RR with laboratory SCL Paris (2007, 2008, 2010).

6 Œnology RR with «Union des Œnologues de France» (UŒF) since UŒF is accredited by COFRAC since RR with the Agricultural Chamber of Gironde since Petrol RR with Total (2009) ELISA RR Mucosal Disease with veterinary public laboratory in Rennes (2008). RR Hypodermasis with veterinary public laboratory in Dijon since Cork in wine RR with the Agricultural Chamber of Gironde since Water in river RR with SATESE Adour-Garonne since COFRAC Auditor COFRAC Auditor as statistical expert (COFRAC = French Committee of Accreditation) 2.3 Training for SCL (customs) laboratories Statistics, uncertainty of measurement, validation of method. General Direction (March and May 2009, May 2008) Rennes (2009) Marseille (2008, 2001) Bordeaux (between 2000 and 2008) Lyon (2006) Lille (2006, 2002) Paris (2006) 2.4 Other training For medical laboratories For veterinary public laboratories For œnology laboratories For other laboratories