1 INTRODUCTION TO WIOA CORE PERFORMANCE INDICATORS: Measurement and Data Collection Presented at: NINAETC Marksville, LA April 2018 Ron D Amico Social Policy Research Associates Oakland, CA
WORKSHOP GOALS 2 Explain key terms Explain the WIOA measures and give examples Review timeline for implementation Discuss observations and implications 2
WHERE TO START? 3 3
Who says so? 4 Authority in the WIOA law Section 116(b) defines the core measures The INA program is subject to the core measures [WIOA Sections 166(e) and 166(h)] WIOA Section 166(i) allows INA grantees to request waivers from most statutory requirements Guidance in the Final Rule and TEGLs Additional guidance on the measures is provided in the regulations ETA issued TEGLs providing clarification (see TEGL 10 16 Ch 1) 4
WIOA IMPLEMENTATION TIMELINE 5 PY 2014 PY 2015 PY 2016 PY 2017 PY 2018 INA performance & reporting requirements expected to go into effect Most provisions take effect (except performance) 7/1/15 WIOA signed into law 7/22/14 New reporting requirements are approved (6/29/16) Performance requirements go into effect (except INA) (7/1/16) TEGL 10-16 Change 1 clarifies the core measures (8/23/17) 5
IMPORTANT NOTE 6 Further guidance specific to the INA program will be forthcoming The WIOA measures are to take effect for PY 2018, but only to gather baseline data New reporting system (to replace Bear Tracks) has been developed and will roll out soon. 6
Don t worry! Performance has been very good 7 Percent of Grantees Meeting PY 16 WIA EER Performance 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% Missed minimum standard 2% Up to 20 pts above minimum standard 15% 20 to 35 pts above minimum standard More than 35 pts above minimum standard 39% 44% 7
KEY DEFINITIONS 8 Participant Date of Participation Qualifying Service Exit Date Exit Quarter 8
PARTICIPANT 9 Eligible for INA Program Receives a Qualifying Service No change from WIA 9
DATE OF PARTICIPATION 10 Eligible Received a qualifying service Date of participation is the date the participant got the first qualifying service No change from WIA 10
QUALIFYING SERVICE 11 Qualifying services include (among other things) Initial and comprehensive assessments Career counseling Development of employability plan Individualized assistance with job search Prevocational services & work experience Training Does not include Eligibility determination Self services Providing general information Follow up services after exit For the INA program, no change from WIA 11
NEW NAMES FOR SOME SERVICES 12 WIA CORE WIA Intensive WIOA Career Services WIA core & intensive services combined into WIOA career services 12
DATE OF EXIT 13 The exit date is the last date the individual received a Qualifying Service. Exit must occur when: - The participant has not received a Qualifying Service for 90 days, and - No further Qualifying Services are planned (i.e., there is no planned gap) BearTracks automatically exits someone with no service after 90 days, unless there is a planned gap Exit date is the date of last service, not the end of the 90 day period No change from WIA 13
14 MEASUREMENT ISSUES 14
EXCLUSIONS 15 Participants are excluded from the measures if, at exit, the following applies: Medical condition expected to last more than 90 days causes the person to exit the program and precludes employment Incarcerated Hospitalized or in other treatment facility Deceased Reservists called to active duty 15
CALENDAR QUARTERS 16 Exits and employment outcomes are measured in calendar quarters Jan Mar Jul Sept Apr June Oct Dec The exit quarter is the calendar quarter in which the exit date falls No change from WIA 16
EXAMPLE 17 The Scenario Eligibility determination is April 21 st First service (assessment) is provided May 4 th Last service (job referral) is provided Sept 4 th No further services are provided for the next 90 days Customer gets a job December 15 th What Is The date of participation? The date of exit? the exit quarter? 17
EXAMPLE 18 The exit quarter is the quarter in which the date of exit occurs Apr May June July Aug Sept Oct Nov Dec Eligibility 4/21 Period of participation 90 days with no service Got a job 12/15 Date of first service 5/4 Exit Date (date of last service) 9/4 End of 90-days 12/2 Exit Quarter 7/1 to 9/30 18
MEASUREMENT QUARTERS 19 Employment outcomes are measured in the 2 nd and 4 th quarters after the exit quarter Exit date 9/4/17 CY 2017 CY 2018 JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN JUL AUG SEP Exit Quarter 2 nd Qtr After 4 th Qtr After 19
HOW EMPLOYMENT OUTCOMES ARE MEASURED 20 Unemployment Insurance (UI) wage matching and grantee data can both be used to measure employment and earnings UI/Federal Wage matching Grantees submit SSNs of participants in the SPIR DOL matches against UI wage files and federal employment database, which contain employment records for (nearly) everyone employed in the U.S. Supplemental (grantee) data Data provided by the grantee through its own follow up 20
WHY USE CALENDAR QUARTERS? 21 Performance is measured using UI wage records. UI records are reported quarterly, and include earnings for the quarter. They usually cannot identify the date of employment, hours of employment, or hourly wages Use of UI data requires a lag of 2 quarters before data become available 21
TYPES OF SUPPLEMENTAL DATA 22 Documentation for supplemental data (data provided by the grantee) include: Pay stubs Letter from an employer Follow up survey of the participant Detailed case notes signed by the counselor If supplemental data is being used, the same source must be used for measuring employment and wages See TEGL 26 16 22
What difference does each data source make? 23 PY 16 Performance on the WIA Measures Grantee Data Only UI Data Only Both EER 43.4% 56.1% 68.4%* Retention 78.1% 75.6% 80.7%* Avg Earnings $10,416 $9,897* na * This is the official performance in WIA 23
24 THE MEASURES 24
WIA and WIOA MEASURES COMPARED 25 WIA Measures WIOA Measures 1. Employed in the 1 ST quarter after exit 2. Retention (employed in the 2 nd and 3 rd quarters after exit) 3. Average earnings in the 2 nd and 3 rd quarters after exit 1. Employed in the 2 nd quarter after exit 2. Employed in the 4 th quarter after exit 3. Median earnings in the 2 nd quarter after exit 4. Credential attainment 5. Skill gains 6. Service to employers 7. INA-specific measure(s) as determined 25
Measure 1: 2 ND QUARTER EMPLOYMENT RATE 26 DEFINITION: The percent of exiters who were employed anytime in the second quarter after the exit quarter Notes Number employed in second quarter after exit quarter All exiters in that exit quarter (except global exclusions) Unsubsidized employment anytime in the second quarter counts Evidence can come from UI/Federal employment data, or supplemental data Not the same as the WIA Entered Employment Rate 26
2 ND QUARTER EMPLOYMENT RATE EXAMPLE 27 Based on our example Exit date 9/4/17 CY 2017 CY 2018 JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN JUL AUG SEP Exit Quarter 2 nd Qtr After 27
2 ND QUARTER EMPLOYMENT RATE QUIZ 28 Scenario First service is May 4 th Last service (job referral) is August 18 th Customer gets a job Sept 15 th Customer loses job Dec 15 th Customer finds another job April 10 th What is The measurement quarter for the employment rate? Is the customer a positive on this outcome? 28
2 ND QUARTER EMPLOYMENT RATE QUIZ 29 Exits 8/18 Gets job 9/15 Loses job 12/15 Gets another job 4/10 29
Measure 2: 4 th QUARTER EMPLOYMENT RATE 30 DEFINITION: The percent of exiters who were employed anytime in the fourth quarter after the exit quarter Number employed in fourth quarter after exit quarter All exiters in that exit quarter (except global exclusions) Notes Unsubsidized employment anytime in the fourth quarter counts Evidence can come from UI/Federal employment data, or supplemental data 30
4 th QUARTER EMPLOYMENT RATE EXAMPLE 31 Based on our example Exit date 9/4 CY 2017 CY 2018 JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN JUL AUG SEP Exit Quarter 4 th Qtr After 31
Measure 3: MEDIAN EARNINGS 32 DEFINITION: The median quarterly earnings of those employed in the second quarter after the exit quarter Notes Measured only for those employed anytime in the 2 nd quarter Based on UI/Federal employment data or supplemental data If you use supplemental data to count someone as employed in the 2nd quarter, you must also measure that person s earnings Differs from WIA Average Earnings - Measurement quarter is different - WIOA measures median earnings, not average earnings - WIOA includes earnings measured from supplemental data 32
MEDIAN EARNINGS EXAMPLE 33 Based on our example Exit date 9/4 CY 2017 CY 2018 JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN JUL AUG SEP Exit Quarter 2 nd Qtr After 33
CALCULATING A MEDIAN 34 Earnings in 2 nd Quarter Person 1 $3,000 Person 2 $4,400 Person 3 $5,400 Person 4 $6,000 Person 5 $7,000 Person 6 $8,200 Person 7 $8,400 Sum $42,400 } Half of people below midpoint Midpoint } Half of people above midpoint Median= $6,000 (the earnings of the person at the midpoint) Average = 42,400 = $6,057 7 34
35 Scenario 1 MEDIAN MINIMIZES THE EFFECT OF EXTREME VALUES Scenario 2 Person 1 $3,000 Person 2 $4,400 Person 3 $5,400 Person 4 $6,000 Person 5 $7,000 Person 6 $8,200 Person 7 $8,400 Sum $42,400 Median= $6,000 (the midpoint) Midpoint Person 1 $3,000 Person 2 $4,400 Person 3 $5,400 Person 4 $6,000 Person 5 $7,000 Person 6 $8,200 Person 7 $15,500 Sum $49,500 Median= $6,000 (the midpoint) Average = 42,400 = $6,057 7 Average = 49,500 = $7,071 7 35
CALCULATING A MEDIAN WITH AN EVEN NUMBER OF PARTICIPANTS 36 Earnings in 2 nd Quarter Person 1 $3,000 Person 2 $4,400 Person 3 $5,400 Person 4 $6,000 Person 5 $7,000 Person 6 $8,200 Person 7 $8,400 Person 8 $8,500 } Middle two people Median= (6000 + 7000) =$6,500 2 36
Measure 4: CREDENTIAL RATE 37 DEFINITION: Percent of those enrolled in education or training (excluding OJT & customized training) who, during participation or within one year after exit date, obtain: postsecondary credential, or high school diploma or equivalent (but only if they are employed or in education or training leading to a recognized post secondary credential within the year) 37
CREDENTIAL RATE 38 Who is in the base? Exiters (except global exclusions) who, while a participant: Were in postsecondary education or training (other than OJT or customized training), or Were in secondary school (don t already have a diploma) Who counts as a success? Those who, during participation or within one year after exit Obtain a postsecondary credential, or Obtain a secondary school diploma or equivalent, but only if they are also employed or enrolled in postsecondary education leading to a credential anytime within the year after exit 38
TYPES OF CREDENTIALS 39 Recognizes attainment of measurable technical or occupational skills necessary to obtain employment or advance Educational GED High school diplomas Associate or Bachelor Degree Industry Recognized Certificates Microsoft Certified IT Professional Auto Repair (ASE) Registered Apprenticeship Machinist Construction trades Baker State Occupational Licenses Cosmetology Teacher Gaming 39
Types of Credentials 40 Credentials do not include: Work readiness certificates Credentials awarded by grantees or state or local workforce boards (unless the grantee is a tribal college or other approved entity) Attainment of general skills related to safety, hygiene, etc., even if required to qualify for employment (example: OSHA 10) Certificates of completion 40
TESTING OUR KNOWLEDGE: EXAMPLE 1 41 Scenario Enrolls April 2017. Chris was a high school student in danger of dropping out. She enrolled in WIOA and the program provided tutoring and other dropout prevention activities. She also participated in work experience during the summer. Exits Nov 2017. By the fall, she was doing much better in school, and she exited the WIOA program in November 2017. What happens next? Chris graduated and got a diploma in May 2018 Got a job in June 2018. Unfortunately, she got laid off in Sept 2018. What are Chris s outcomes? Is she a positive on the credential attainment rate? What other outcomes might she achieve? 41
TESTING OUR KNOWLEDGE : EXAMPLE 1 42 Measurement period for obtaining a credential is date of participation to 1 year after exit date. It is not tied to calendar quarters Enrollment date 4/15 CY 2017 Exit date 11/28 Got a job Graduated 6/20 5/25 CY 2018 Lost job 9/18 APR MAY JUN JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN JUL AUG SEP Oct Nov Dec Jan Period of Participation Exit Qtr 2nd Qtr 4th Qtr Year After Exit Date Measurement period for obtaining a credential is 4/15/16 to 11/17/17 42
Measure 5: SKILL GAIN 43 DEFINITION: Percent of participants who, during the course of the program year, are in education or training programs that lead to a recognized postsecondary credential or employment, and who are achieving measurable skill gains, defined as documented academic, technical, occupational, or other forms of progress, towards such a credential or employment 43
SKILL GAIN continued 44 Designed to measure interim progress of participants, as well as those who exited during the reporting period Key Elements Base includes those who, during the reporting period, were in education or training programs that lead to a recognized postsecondary credential or employment Includes those in secondary school, college, or other training (including those in OJT) Does not include work experience (this is not training, according to WIOA) Each participant can count as a success only once during a reporting period, even if the individual has more than one skill gain Participants can count in multiple reporting periods if they have at least one skill gain in each period 44
SKILL GAIN continued 45 Indicators of Skill Gains Documentation of skill gain includes: Increasing at least one educational functional level (for those in programs below the postsecondary level) Attainment of high school diploma or GED Making satisfactory progress in school (e.g., earning sufficient credit hours) Secondary school: transcript or report card for 1 semester Postsecondary: at least 12 credit hours in a semester or, for part time students, 12 credit hours over 2 consecutive semesters Making satisfactory progress towards measurable standards as attested by an employer or training provider Passing an exam designed to measure occupational or technical skills 45
MEASURE 6: EFFECTIVENESS IN SERVING EMPLOYERS 46 Multiple approaches being considered, but INA program will use Retention with Same Employer Number employed in the 4 th quarter with the same employer as in the 2 nd quarter Number employed in the 2 nd quarter after exit quarter Logistics Easiest to measure: would be calculated strictly through UI wage matching Does not require additional data collection by the grantee Could not be calculated until the 4 th quarter after exit 46
Measure 7. INA SPECIFIC MEASURE 47 Rationale INA program is to promote the economic and social development of Indian, Alaska Native, and Native Hawaiian communities in accordance with (their) goals and values. Authority According to WIOA law, ETA, in consultation with the Advisory Council, must develop a set of performance indicators and standards that are applicable to the INA program. 47
Measure 7. INA SPECIFIC MEASURE 48 Progress to Date DINAP has requested that ETA accept waivers from grantees for: Measurable Skills Gains 4 th Quarter Employment Rate To be replaced with: Employability Enhancement Rate Percent in Training This proposal is still being discussed in the Department and has not yet been approved 48
SETTING STANDARDS 49 Using an Adjustment Model For the State programs, standards should be set using a statistical model that takes into account: o o Economic conditions Characteristics of participants Performance standards for the Section 166 program should use to the extent practicable, the statistical adjustment model used for states [WIA Section 166(h)] 49
50 THE YOUTH MEASURES
WIOA MEASURES 51 CSP Measures 1. Employed in the 2 nd quarter after exit 2. Employed in the 4 th quarter after exit 3. Median earnings in the 2 nd quarter after exit 4. Credential attainment 5. Skill gains 6. Service to employers 7. INA-specific measures, as determined Youth Measures 1. Employed or in education or training in the 2 nd quarter after exit 2. Employed or in education or training in the 4 th quarter after exit 3. Median earnings in the 2 nd quarter after exit 4. Credential attainment 5. Skill gains 6. Service to employers 7. INA-specific measures, as determined 51
52 OBSERVATIONS & NEXT STEPS
OBSERVATIONS 53 Statutory Language Limits DINAP s Options Some Important Flexibility Legislation requires use of the core measures for the INA program Grantees can seek waivers Advisory Council has helped develop INA specific measures DINAP is seeking preapproval for waivers 53
IMPLICATIONS FOR REPORTING 54 Changed reporting forms: 9084 & 9085 will be replaced PIRL replaces the SPIR New forms have already been approved by DOL and OMB
NEW DATA SYSTEM 55 Bear Tracks is being replaced Changes Eliminates concepts of core and intensive services Collects data for new outcomes Will need to submit data on participants, not just exiters Features Will be a cloud based system Already developed for CSP program: Launch Date: July 1, 2018 Database for Youth Program will follow later 55
WHAT TO EXPECT 56 PY 2018 is likely to be a transition year No standards set, but baseline data collection likely to begin Bear Tracks SPIR will be examined for what it reveals - Data on employment in the 2 nd quarter - Wages - Credential attainment New cloud-based system will be phased in Advisory Council will continue to advocate for additional flexibility Additional guidance likely to be forthcoming from DINAP 56
WHAT TO DO? 57 What to do in the meantime Keep the focus on serving your customers and community Begin planning for new data collection - Credential Attainment & Measurable Skill Gains - Collect supplemental data on employment Make better use of follow-up services - Think about allocating budget for follow-up services - Supportive services and counseling are allowed for up to 12 months after exit (see Final Rule 684.310(d)) Support your Advisory Council - Make your challenges and needs known - Seek as much flexibility as allowable under the law 57
RESOURCES 58 Final Rule for ETA (issued 8/19/16) Provides regulations for ETA only WIOA provisions Regulations for the INA program at Section 684 Available from the Federal Register at https://www.gpo.gov/fdsys/pkg/fr 2016 08 19/pdf/2016 15975.pdf Final Joint Rule (issued 8/19/15) Describes performance standards requirements (see Section 677) Available from the Federal Register at https://www.gpo.gov/fdsys/pkg/fr 2016 08 19/pdf/2016 15977.pdf Approved Information Collection Request Describes proposed new individual level record layout (to replace the SPIR) and new aggregate report Available at https://doleta.gov/performance/reporting/eta_default.cfm
2 ND QUARTER EMPLOYMENT RATE QUIZ 59 Exits 8/18 Gets job 9/15 Loses job 12/15 Gets another job 4/10 Exit Quarter 2 nd Qtr After Exit Employed 9/15 to 12/15 Not Employed Employed 4/10 to 59
60 TESTING OUR KNOWLEDGE: EXAMPLE 1 Answer Chris gets credit as a success on the Credential Attainment Rate High school diploma is an allowable credential type She obtained his diploma sometime in the measurement period To get credit for a high school diploma, you must be employed sometime in the year after the exit date, and she was Chris gets credit as a success on the 2 nd Quarter Employment Rate She was employed sometime in the second quarter after the exit quarter She is included in the calculations of Median Earnings (but has very low quarterly earnings, since she was only employed 10 days in the quarter) She is a negative on the 4th Quarter Employment Rate 60