1 INTRODUCTION TO WIOA COMMON PERFORMANCE MEASURES WORKSHOP GOALS 2 Review Key Definitions Highlight Differences between WIOA and WIA Measures Share Observations and Implications for Reporting 2 FINAL WIOA REGULATIONS COMING SOON 3 Information in this workshop taken from WIOA law and interim regulations. Final regulations will further clarify definitions and measures. 3 1
KEY DEFINITIONS 4 4 SAME DEFINITIONS WIA AND WIOA 5 Participant Date of Participation Service Exit Exit Quarter 6 Planned Gap Global Exclusions NOT YET DEFINED UNDER WIOA 2
PARTICIPANT 7 Eligible for WIOA Receives a WIOA Service 7 DATE OF PARTICIPATION 8 Eligible Received a WIOA service If both checked, it s the date the participant received the WIOA service. 8 SERVICE 9 Service includes: In program WIOA activities, such as career services and training services. Significant staff involvement, e.g. assessment of a participant s skills, education, or career objectives 9 3
SERVICE IS NOT 10 Service does not include: Determination of eligibility Services and activities specifically provided as followup services Regular contact with the participant or employer only to obtain information regarding employment or educational status or the need for support services Self Service activities Informational activities (e.g. LMI) 10 DATE OF EXIT 11 The last day on which the individual received a service funded by WIOA or a partner program. 11 EXIT 12 BearTracks will automatically exit a participant who has not received a WIOA service within 90 days and is not scheduled for future services. 12 4
QUARTERS 13 Quarter 1 Quarter 2 Quarter 3 Quarter 4 JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC 13 Quarters QUARTERS 14 Most performance measures are calculated based on the quarter in which a participant exits, and the performance outcomes are due in different quarters. 14 EXAMPLE QUARTERS 15 EXIT EXIT JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC 1 st Quarter 2 nd Quarter after Exit after Exit JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC 3 rd Quarter 4 th Quarter after Exit after Exit 15 5
WHY USE CALENDAR QUARTERS? 16 Performance is measured using Unemployment Insurance (UI) wage records. UI records are reported quarterly, and include earnings for the quarter. They usually cannot identify the date of employment, hours of employment, or hourly wages. 16 GLOBAL EXCLUSIONS* 17 Participants are excluded from the common measures for the following reasons: Institutionalized Invalid or missing SSN Health/Medical or Family Care Reservists called to active duty Deceased *Not yet defined under WIOA 17 GLOBAL EXCLUSIONS* 18 Global Exclusions can be taken until the third quarter after exit. *Not yet defined under WIOA 18 6
PLANNED GAP* 19 Participants should not be considered as exited if there is a planned gap in service of greater than 90 days in one of the following circumstances: Health/medical condition or providing care for a family member with a health/medical condition Delay before the beginning of training Temporary move from the area that prevents the individual from participating in services. *Not yet defined under WIOA 19 20 THE MEASURES WIOA MEASURES 21 Adult Measures 1. Employed in the 2 nd quarter after exit 2. Employed in the 4 th quarter after exit 3. Median earnings in the 2 nd quarter after exit 4. Credential attainment 5. Skill gains 6. Service to employers 7. Other measures as determined Youth Measures 1. Employed or in education or training in the 2 nd quarter after exit 2. Employed or in education or training in the 4 th quarter after exit 3. Median earnings in the 2 nd quarter after exit 4. Credential attainment 5. Skill gains 6. Service to employers 21 7
22 WHY ARE THEY COMMON? In addition to Indian Programs, these programs share WIOA common measures, among others: Title I Adult, Dislocated Worker and Youth programs Adult Education and Literacy programs Wagner Peyser Employment Service Title I of the Rehabilitation Act programs Job Corps, YouthBuild, and Migrant and Seasonal Farmworker programs Department of Education OCTAE and OSERS programs USDA programs Health and Human Services programs 1. 2 ND QUARTER EMPLOYMENT RATE 23 The percent of exiters who were employed anytime in the second quarter after the exit quarter. 23 EMPLOYMENT RATE COMPARISON 24 WIA Entered Employment Rate (EER): Of those not employed on the date of participation, the percent who were employed anytime in the first quarter after the exit quarter WIOA Second Quarter Employment Rate: The percent of exiters who were employed anytime in the second quarter after the exit quarter Difference Measures employment in the second quarter, not the first quarter after exit All exiters are included in the base (does not exclude those not employed on the date of participation) 24 8
2 ND QUARTER EMPLOYMENT RATE EXAMPLE 25 Exit/ Job Loses Job Gets Job JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC 25 2 ND QUARTER EMPLOYMENT RATE EXAMPLE 26 Exit/ Job Loses Job Gets Job JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC 2 nd Quarter after exit 26 2 ND QUARTER EMPLOYMENT RATE EXAMPLE 27 Exit Gets Job Loses Job JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC 2nd Quarter after exit 27 9
2. 4 TH QUARTER EMPLOYMENT RATE 28 The percent of exiters who were employed anytime in the fourth quarter after the exit quarter. 28 SUBSEQUENT EMPLOYMENT RATE COMPARISON 29 WIA Retention Rate: Of those employed in the first quarter after exit, the percent who were employed anytime in both the second and third quarters after the exit quarter WIOA Fourth Quarter Employment Rate: The percent of exiters who were employed anytime in the fourth quarter after the exit quarter Difference Not a retention rate, because it does not restrict the base to those employed in an earlier quarter Measures employment in the fourth quarter, not the second and third quarter after exit 29 4 TH QUARTER EMPLOYMENT RATE EXAMPLE 30 EXIT/ Job Loses Job JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC Gets Job JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC 4 th Quarter after Exit 30 10
3. MEDIAN EARNINGS 31 The median quarterly earnings of those employed in the second quarter after the exit quarter. 31 EARNINGS COMPARISON 32 WIA Average Earnings: Of those employed in the first and second and third quarters after exit, earnings in the 2nd plus 3rd quarters after the exit quarter divided by those in the base WIOA Median Earnings in the Second Quarter:The median quarterly earnings of those employed in the second quarter after the exit quarter Difference Measures earnings in the second quarter, not the second and third quarters Does not require employment in all three quarters Calculates median earnings, not the average earnings 32 MEDIAN EARNINGS EXAMPLE 33 JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC Enters EXIT JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC Post-Program JANEarnings FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC 33 11
CALCULATING A MEDIAN 34 Earnings in 2 nd Quarter Person 1 $1,500 Person 2 $2,200 Person 3 $2,700 Person 4 $3,000 Person 5 $3,500 Person 6 $6,200 Person 7 $8,200 Sum $27,300 Average = 27300 = $3,900 7 } Half of people below midpoint Midpoint } Half of people above midpoint Median= $3,000 (the earnings of the person at the midpoint) 34 MEDIAN MINIMIZES THE EFFECT OF EXTREME VALUES 35 Earnings in 2 nd Quarter Person 1 $1,500 Person 2 $2,200 Person 3 $2,700 Person 4 $3,000 Person 5 $3,500 Person 6 $6,200 Person 7 $14,000 Sum $33,100 Average= 33100 = $4,729 7 } Half of people below midpoint Midpoint } Half of people above midpoint Median= $3,000 (the midpoint) 35 CALCULATING A MEDIAN WITH AN EVEN NUMBER OF PARTICIPANTS 36 Earnings in 2 nd Quarter Person 1 $1,500 Person 2 $2,200 Person 3 $2,700 Person 4 $3,000 Person 5 $3,200 Person 6 $3,500 Person 7 $6,200 Person 8 $8,200 } Median= (3000 + 3200) =$3,100 2 Middle two people 36 12
4. CREDENTIAL RATE 37 Percent of exiters who obtain a post secondary credential or high school diploma or equivalent by the 4th quarter after exit. Those obtaining a high school diploma or equivalent count as a success only if they are employed or in education or training leading to a recognized postsecondary credential within the year. 37 CREDENTIAL RATE COMPARISON 38 WIA Credential Rate: Percent of exiters who obtain a credential while a participant or up to three quarters after the exit quarter. Credentials include high school diplomas (or equivalents), post secondary degrees, recognized certificates, or occupational licenses. WIOA Credential Rate:Percent of exiters who obtain a post secondary credential or high school diploma or equivalent by the 4 th quarter after exit. Those obtaining a high school diploma or equivalent count as a success only if they are employed or in education or training leading to a recognized postsecondary credential within the year. Difference This was just a pilot for INA program under WIA Can be obtained up to four quarters after exit, not just three quarters Does not count high school diplomas or equivalents unless the person was employed or in other training Unclear if the definition of credential will change 38 CREDENTIAL RATE EXAMPLE 39 Enter Training Finish Training Licensing Exam EXIT JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC 1 st Quarter after 2 nd Quarter after Exit Exit JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC 3 rd Quarter after 4 th Quarter after Exit Exit 39 13
5. SKILL GAIN 40 Percent of participants who, during the course of the program year, are in an education or training program and are achieving measurable skill gains, defined as documented academic or other progress. 40 SKILL GAIN COMPARISON 41 WIA Not a WIA measure WIOA Skill Gain:Percent of participants who, during the course of the program year, are in an education or training program and are achieving measurable skill gains, defined as documented academic or other progress. 41 SKILL GAIN continued 42 Discussion Designed to measure interim progress of participants Documentation might include: Increasing at least one educational functional level for those in ABE programs Attainment of high school diploma or GED Making satisfactory progress in school Passing an exam designed to measure skills Measurable observable performance based on industry standards Unclear what the time interval is (e.g., is it measured every program year for every participant? Does it matter what the date of enrollment was?) Specifics are still up in the air 42 14
6. SERVING EMPLOYERS 43 Effectiveness in serving employers. 43 6. SERVING EMPLOYERS 44 WIA Not a WIA measure WIOA Effectiveness in serving employers. 44 6. SERVING EMPLOYERS continued 45 Discussion Should capture effectiveness while minimizing burden on employers Possible approaches being considered Employee retention rates of WIOA participants placed in jobs Percent of employers who are repeat customers Percent of all employers in the service area who use WIOA services Should consider joint effectiveness of multiple program partners Specifics are still up in the air 45 15
7. INDIAN PROGRAMS SPECIFIC MEASURE(S) 46 WIA No specific measure, but ability to effectively serve native peoples and communities has always been understood as a critical part of grant responsibility WIOA Discussion In addition to the required measures, ETA, in consultation with the Advisory Council, must develop a set of performance indicators and standards that are applicable to the INA program. ETA hasn t worked with the Advisory Council to develop these measures yet This is a statutory requirement, so work on this should start soon 46 47 THE YOUTH MEASURES 1. EMPLOYED/ PLACED AFTER EXIT 48 The percent of exiters who were employed or in education or training during the second quarter after exit 48 16
EMPLOYED/PLACED AFTER EXIT COMPARISON 49 WIA Not on the menu of measures WIOA Discussion Second Quarter Placement Rate: The percent of exiters who were employed or in education or training during the second quarter after exit Measured as of the second quarter after exit Counts unsubsidized employed or participation in any education or training (e.g., secondary, postsecondary, or other) Calculated as a percent of all youth exiters 49 2. SUBSEQUENT EMPLOYMENT/ PLACEMENT 50 The percent of exiters who were employed or in education or training during the fourth quarter after exit 50 2. SUBSEQUENT EMPLOYMENT/ PLACEMENT COMPARISON 51 WIA Not on the menu of measures WIOA Discussion Fourth Quarter Placement Rate: The percent of exiters who were employed or in education or training during the fourth quarter after exit Measured as of the fourth quarter after exit Counts unsubsidized employed or participation in any education or training (e.g., secondary, postsecondary, or other) Calculated as a percent of all youth exiters 51 17
OTHER WIOA YOUTH MEASURES 52 The remaining youth measures are identical to adult program measures just discussed: 3. Median Earnings 4. Credential Attainment Rate 5. Skill Gain 6. Service to Employers 52 HOW OUTCOMES ARE MEASURED 53 UI Wage Matching AND Grantee Data UI Wage Matching Grantees submit SSNs of participants in the SPIR each quarter DOL matches against Unemployment Insurance (UI) wage files, containing a record of (nearly) everyone employed in the U.S. Supplemental Data Provided by Grantees Also as part of the SPIR, grantees indicate which participants are employed according to the grantee's own records 53 HOW OUTCOMES ARE MEASURED 54 Result? Both sources are used for 2 nd and 4 th Quarter Employment Likely Only UI data to be used for measuring Median Earnings Use of UI data implies a lag of 5 quarters before outcomes are measured for a person who exits Three quarters due to the measurement quarters UI data are available with an additional 2 quarter lag 54 18
SETTING STANDARDS 55 Using an Adjustment Model For the State programs, standards should be set using a statistical model that takes into account: o Economic conditions o Characteristics of participants Performance standards for the Section 166 program should use to the extent practicable, the statistical adjustment model used for states [WIA Section 166(h)] 55 OBSERVATIONS 56 Implementation Fallen Behind Schedule Likely not possible to fully implement by 7/1/16 Statutory Language Limits DINAP s Options Legislation requires use of common measures for program Some Important Flexibility Grantees can seek waivers Advisory Council can help develop INAP specific measures 56 MORE OBSERVATIONS 57 In some ways the WIOA measures are more straightforward Most measures are calculated as a percent of all exiters/participants, rather than subsets Follow up services will be key for program and participant success Think about budget allocation for follow up services; beyond merely contacting people for employment and education status Need better real time measures of success 57 19
MORE OBSERVATIONS 58 However, there are significant challenges: For program management, it is harder to time exits to maximize success on Employment measure Measuring as far out as 4 quarters after exit is more of a challenge Measuring skill gains could be difficult Measures for the Supplemental Youth program are especially challenging, given the meager funding Lots of uncertainty remains (e.g., services to employers) 58 IMPLICATIONS FOR DATA COLLECTION & REPORTING 59 Bear Tracks will need to be modified or replaced o Concepts of core and intensive services no longer apply o New outcomes need to be reported o Some changes to info on participant characteristics is required (but not many) o May need to provide data on participants, not just exiters 59 60 IMPLICATIONS FOR DATA COLLECTION & REPORTING continued Requires changes to the: 9084 9085 SPIR 20
IMPLICATIONS FOR DATA COLLECTION & REPORTING continued 61 Requires new data collection for grantees o Need follow up for as long as four quarters after exit o Follow up needed for more than just employment Credential attainment Placement in education or training (for Supplemental Youth participants) o Need to measure additional outcomes UI wage matching through CRIS becomes more valuable DISCUSSION 62 RESOURCES 63 Proposed Rule for ETA Issued 4/16/15 Provides proposed regulations for ETA only WIOA provisions Regulations for the INA program at Section 684 Available from the Federal Register at https://www.gpo.gov/fdsys/pkg/fr 2015 04 16/pdf/2015 05530.pdf Proposed Rule w/ Dept of Education Issued 4/16/15 Describes performance standards requirements (see Section 677) Available from the Federal Register at https://www.gpo.gov/fdsys/pkg/fr 2015 04 16/pdf/2015 05528.pdf Information Collection Request for Comment Issued 9/1/15 Describes proposed new individual level record layout (to replace the SPIR) and new aggregate report Available from Regulations.Gov at: http://www.regulations.gov/#!docketdetail;d=eta 2015 0008 21
LEGISLATIVE AUTHORITY 64 The common measures are defined in Section 116(b) of WIOA. Indian and Native American Program is subject to the common measures, as well as other measures developed in consultation with the Advisory Council [WIOA Sections 166(e) and 166(h)] Additional guidance on the measures is provided in the draft regulations: o 20 CFR 684.620 o 20 CFR 677, including preamble (p. 20586) WIOA Section 166(i) allows INA grantees to request waivers from most statutory requirements CONTACT INFORMATION 65 Laura Aron Senior Associate Social Policy Research Associates 510-788-2476 Laura@spra.com 22