Applied Behavior Analysis Technician (ABAT ) Pass Point Study Data Analysis Report

Transcription

1 Applied Behavior Analysis Technician (ABAT ) Pass Point Study Data Analysis Report Tina Freilicher, Ph.D., Shoreline Psychometric Services, LLC. NOTE: This report describes the analysis of data collected from a cut score study that was conducted by Innovative Learning for the Applied Behavior Analysis Technician (ABAT ) Examination.

2 Table of Contents Introduction... 3 Descriptive Statistics of Judges Ratings... 3 Statistical Summary of Ratings... 4 Table 1 - Summary of Judges' Ratings... 3 Table 2 - Summary Statistics... 4 Table 3 - Ratings by Item and Judge... 4 Shoreline Psychometric Services, LLC, 8/29/2015 2

3 Introduction Innovative- Learning LLC. conducted a cut score study on the Applied Behavior Analysis Technician (ABAT ) Examination. Shoreline Psychometric Services, LLC. was not involved with the conduct of the study but was retained by Innovative- Learning, LLC. to analyze the data produced from the cut score study. This report provides analysis of the data produced from the cut score study. To conduct these analyses, Innovative- Learning provided Shoreline Psychometric Services, LLC. with the judges Angoff ratings in an Excel spreadsheet. Descriptive Statistics of Judges Ratings The spreadsheet contained the ratings made by 6 judges on the 68- item examination. The columns represented the individual judge s ratings, and the rows represented the items on the exam. The file also included average ratings by judge, the standard deviation of the ratings by item, and the pass point (i.e., 72.94). There were no missing ratings. The file contained the names of the judges; however, the names were removed and were numbered (i.e., Judge 1, Judge 2, etc.) when producing the data tables for this report. A quality control check was performed on the calculations of the average rating by judge, the standard deviation of the ratings by item, and the pass point. The quality control check indicated that the average ratings by judge were calculated based on 66 items and not 68 items. The formula used to calculate the average ratings by judge did not include the last two rows of ratings (i.e., ratings for items 67 and 68). In addition, the pass point was calculated without the last two rows of data, which yielded a pass point of The standard deviations of the ratings by item were correct. It appears that the miscalculation was an oversight, as there is no indication that the items were intentionally removed from the analysis. Therefore, the average ratings of judges and the pass point that are presented in this report were calculated using the ratings of the 68 items. Table 1 presents a summary of the judges ratings. For each judge, presented are the number of items (N), the minimum and maximum ratings, average (M) rating, the standard error of the mean (Std. Error), and the standard deviation (SD). As shown on Table 1, the lowest average rating by judge was for Judge #4, and the highest average rating was for Judge #6. Table 1 - Summary of Judges' Ratings Descriptive Statistics Judge N Minimum Rating Maximum Rating M Std. Error SD Shoreline Psychometric Services, LLC, 8/29/2015 3

4 Statistical Summary of Ratings Table 2 shows the following summary statistics: average rating, SD, minimum mean item rating, maximum mean item rating, standard error of the mean (SE), reliability, standard error of measurement (SEM), maximum number of items, and number of judges. The average rating, or pass point was or 73. Based on classical test theory, using the SEM, the true pass point would range from 70.4 to (i.e., plus or minus 2.38). Table 3 shows the Angoff ratings by item and judge, and the means and standard deviations by item and judge. Table 2 - Summary Statistics Summary Statistics Average Rating Standard Deviation (SD) 6.59 Minimum Mean Item Rating Maximum Mean Item Rating Standard Error of the Mean (SE) 2.69 Inter- rater Reliability (Intra- class Correlation Coefficient).869 Standard Error of Measurement (SEM) 2.38 Maximum Number of Items 68 Number of Judges 6 Table 3 - Ratings by Item and Judge Angoff Ratings by Item and Judge Item # Judge 1 Judge 2 Judge 3 Judge 4 Judge 5 Judge 6 M SD Shoreline Psychometric Services, LLC, 8/29/2015 4

5 Angoff Ratings by Item and Judge Item # Judge 1 Judge 2 Judge 3 Judge 4 Judge 5 Judge 6 M SD Shoreline Psychometric Services, LLC, 8/29/2015 5

6 Angoff Ratings by Item and Judge Item # Judge 1 Judge 2 Judge 3 Judge 4 Judge 5 Judge 6 M SD M SD Shoreline Psychometric Services, LLC, 8/29/2015 6