E B C L. Ii E A U ~ L RB A SURVEY OF ACTUAL TEST-SCORE DISTRIBUTIONS WITH RESPECT TO SKEWNESS AND KURTOSIS. Frederic M~ Lord

Transcription

1 ~ E S E B A U ~ L C L Ii E T I N A SURVEY OF ACTUAL TEST-SCORE DISTRIBUTIONS WITH RESPECT TO SKEWNESS AND KURTOSIS RB Frederic M~ Lord Educational Testing Service Princeton, New Jersey August 1954

2 A SURVEY OF ACTUAL TEST-SCORE DISTRIBUTIONS WITH RESPECT TO SKEWNESS AND KURTOSIS Abstract A survey of 48 different aptitude and achievement tests administered to various populations shows ~hat (a) the easier tests tend to have negatively skewed, and the more difficult tests positively skewed, score distributions; (b) most of the score distributions are at least slightly platykurtic, some of them distinctly so.

3 A SURVEY OF ACTUAL TEST-SCORE DISTRIBUTIONS WITH: RESPECT TO SKEWNESS Ai'ilD KURTOSIS.* Purpose The purpose of the present survey of data was to check empirically on two hypotheses: 1. Easy tests tend to yield negatively skewed score distributions; hard tests~ positively skewed distributions. This hypothesis is so prevalent that references need not be cited. 2. Symmetric score distributions are ordinarily platykurtic. This conclusion has been reached from theoretical considerations by Keats (5) (a symmetric beta function is always platykurtic) and Lord (6). The same conclusion has been reached from practical experience by Mollenkopf (7), by Conrad and by Tucker (both cited in 7, p. 214). The practical advantages of a platykurtic score distribution are cited by Ferguson (2) and by Jackson and Fer'guson (4). It may be noted that these two hypotheses derive from a consideration only of the peculiarities of psychological tests used as measuring instruments; these hypotheses are not related to any supposed peculiarities of the group tested. Data The distributions of raw scores for 48 different tests in about 11 different testing programs of the Educational 'resting Service over a twelve-month period starting February 1952 were selected for study. All the raw~score distributions conveniently at hand for regular *Much of the raw data used for the present study were provided by Miss Frances Swineford, who has made many helpful suggestions and criticisms.

4 -2- testings within this twelve-month period were used in the study providing that the number of cases ( N ) was at least 500, except that parallel or approximately parallel forms of the same test were represented only once. A summary of the 48 tests used is given in Table 1, A frequency distribution of the corresponding 48 values on N is given in Table 2. A 1ittlemore than half of the tests were composed exclusively of 5-choice itemsj the remainder range from 2-choice to 11-choice in various mixtures. Somewhat more than half of the test scores are "corrected for guessing" by subtracting a fraction of the wrong answers from the number right. It would have been desirable from the point of view of the present study for all the tests to have been uniform in the foregoing respects; this was not possible, however, nor did there seem to be any way to make an adequate correction for the differences, which have consequent1y been ignored. Statistical Treatment The following statistics were obtained for each distribution: 1. number of cases ( N ) 2. maximum possible score :3 mean obtained score 4. median ( P50 ) 5. first and ninth deci1es ( P 10 and P ) 90 6 first and third quartiles ( P 25 and P75 ) 8. semi-interquartile range, Q =! ( P 75 - P ) 2 25

5 -3-9 skewness, Sk :::~. ( P 90 + PIa) - P kurtosis, Ku ::: Qjn 11. standard error of skewness? S.E. (sx) :::.5185 DI {N 12. standard error of kurtosis, S.E. (Ku) :::.27779/ {if 13. C.R.(Sk), the ratio of the skewness to its standard error 14. C.R.(Ku)::: (Ku -.263)/S.E.(Ku) 15. skin The formulas 9-14 are taken from (3). Although more efficient f'ormulas may be available for use with di.stributions that are nearly normal, efficiency is not of great importance here in view of the large values of N. The two critical ratios, C.R.(Sk) and C.R.(Ku), provide statistical tests as to whether or not the skewness and kurtosis differ significantly from the skewness and kurtosis of a normal distribution. Results - 1. Skewness Since the values of Sk are expressed in the units of the raw~score scales of the several tests? it seems appropriate to examine the values of Sk!n rather than those of Sk itself. For the 48 tests, these latter values range from -.14 to +.17 Thirty of the tests were. found to be significantly skewed at the 5-per-cent level. (It should be noted that whenn is very large, as is often the case here? a rather small degree of skewness may be found to be significant.' Table 3 is a scatter diagram showing the relation of skin to test difficulty, the latter being measured by the ratio of

6 -4~ mean score to maximum possible score. It is clear that there is a quite considerable relation between the difficulty of the test and the skewness of the resulting distribution of scores, as Specified by the first hypothesis under investigationo Results - 2. Kurtosis The kurtosis of the score distribution was studied for the 18 tests that were found not to have significantly skewed score distributions0 The values of Ku obtained ranged from.256 to.299. Only three of the 18 score distributions were leptokurtic, having a value of Ku less than.263; the critical ratios for these three lay between 0 and-loo The remaining 15 distributions were platykurtic, 7 of them being significantly so beyond the 5-per cent level (one-tailed test). This result confirms the second hypothesis under investigation. Since the meaning of a value of Xu (other than Ku =.26; ) is not readily apparent without previous experience with this statistic, Figure 1. is presented, showing a comparison between one of the two most platykurtic observed score distributions and a normal distribution having the same total frequencyi mean, and standard deviationo The observed score distribution is based on 2,354 cases and has a kurtosis of Ku =.299. The maximum possible score is 600 It is apparent that the deviation of this distribution from normality is appreciable.

7 -5- Summary and Discussion In a survey of 48 different aptitude and achievement tests administered to various populations at the higher educational levels, it is found (a) that the easier tests tend to have negatively skewed, and the more difficult tests positively skewed, score distributions; (b) that 15 of the 18 symmetric score distributions are slightly platykurtic, 7 of them significantly so. Characteristics of the shape of score distributions may be due either to peculiarities of the measuring instrument (the psychological test) or to peculiarities of the group tested. It might have been assumed a priori that most of the groups tested would have positively skewed distributions of ability, since all or most of them are highly selected. Oddly enough, the score distributions show little evidence of a general tendency toward positive skewness such as would have been expected on this account. The observ-ed facts about the score distributions, as highlighted in the two preceding sections, do not seem to correspond to any suspected peculiarity of the groups tested. Consequently, these particular aspects of the observed results may be assumed to be attributable to properties of the measuring instruments, as hypothesized, rather than to the shapes of the distributions of ability in the groups tested. The results relating to skewness merely reconfirm what was already generally known. The results relating to kurtosis require further consideration.

8 -6- In most cases the kurtosis is probably so slight that statistical tests of significance based on the assumption of normality are not much impaired. A relatively small degree of kurtosis can have important repercussions, however, whenever individuals at the extremes of the score distribution are under consideration. In the case of the test shown in Figure 1, no examinee ob tained a score as much as 2 standard deviations above the mean and less than one percent of the examinees obtained scores as 1 much as 2 standard deviations below the mean, whereas 2"2 per cent of the cases are found in each of these extreme areas in a normal distribution. This situation may not be serious as long as the test user has all raw scores in front of him. It does become serious if standard scores, which are merely a linear transformation of the raw scores, are reported with the usual implication that the proportion of cases obtaining a given standard score is approximately the same as would be found for a normal distribution. the user has access 'Ihe difficulty is especially serious if only to one or a few of the scores of the group tested. The foregoing difficulty is readily avoided if normalized scores rather than standardized scores are used, as urged in Technical.Recommendations for Psychological Tests and Diagnostic Techniques (1). No transformation of the raw scores, however, will alter the fact that the very bright (dull) students have been squeezed together near the top (bottom) of the scale, and that the test consequently does not provide good discrimination among them.

9 -7- References 1. American Psychological Association, American Educational Research Association, and National Council on Measurements Used in Education. Technical Recorrunendations for Psychological Tests and Diagnostic Techniques. Supplement to the Psychological Bulletin, vol. 51, 1951~. 2. Ferguson, G. A. On the theory of test discrimination. Psychometrika, 1949, 14, Garrett, H. E. Statistics in psychology and education. (4th ss.) New York: Longmans, Green and Co., Jackson, R. W. B. and Ferguson, G. A. A functional approach in test construction. Educ. psychol. Meas., 1943, 3, Keats, J. A. A statistical theory of objective test scores. Melbourne: Australian Council for Educational Research, Lord, F. M. A theory of test scores. Doctoral dissertation, Princeton University, Also Psychometric Monograph No.7, Mollenkopf, W. G. Variation of the standard error of measurement. Psychometrika, 1949,14,

10 Table 1. List of the 48 Tests Studied Battery or Program Title Navy College Aptitude Test Tests No. of Tests 1 College Entrance Examination Board College Entrance Examination Board Law School Admission Test Verbal, Mathematics 2 English, Social Studies, French, 1; German, Latin,.Spanish" Biology, Chemistry, Physics, Mathematics, Spatial, Pre-Engineering 1 u. S. Department of State Selective Service Qualification Test Medical College Admission Test Preliminary Actuarial Examination u. S. Naval Academy Entrance Examinations West Point Entrance Examinations National Teacher Examinations National Board of Medical Examiners French, Spanish Verbal, Quantitative, Science, Social Science Verbal, Mathematics AptitUde, English, Algebra, Geometry, History Science, English, Mathematics Professional Information, English, Humanities, Science, Nonverbal Reasoning, Education, Social Science Pathology, Pharmaecology, Internal Medicine, Pediatrics, Public Health ;

11 Table 2. Sizes of the Groups Tested with the 48 Tests Si2;e of Group ( N ) Number of Tests 16,000 to 32, ,000 to 16, ,000 to 8, ,000 to 4, ,000 to 2,)' to 1,

12 Table 3. Scatter Diagram between the Skewness of the Score Distribution and Difficulty of the Test Test Difficulty (Mean Score/Maximum Possible Score) Skewness (Sk!D) to to to to ".15 to to to to ~O to ,,,10 to to

13 en (I) (J) o q to 11 o