Washington state workforce data quality initiative An analysis of historical service data from SKIES: to inform data quality improvements Published June 2018 Washington State Cami Feek, Interim Commissioner Workforce Information and Technology Services Cynthia Forland, Ph.D., Chief Information Officer Workforce Information and Technology Services System Performance Unit Sarah McGraw, Economic Analyst Kevin Persell, Economic Analyst Byron Mukai, Economic Analyst This report was prepared in accordance with the provisions of the Workforce Data Quality Initiative (WDQI) Grant and we acknowledge the financial support of the United States Department of Labor. This workforce product was funded by a grant awarded by the U.S. Department of Labor s Employment and Training Administration. The product was created by the recipient and does not necessarily reflect the official position of the U.S. Department of Labor. The Department of Labor makes no guarantees, warranties, or assurances of any kind, expressed or implied, with respect to such information, including any information on linked sites and including, but not limited to, accuracy of the information or its completeness, timeliness, usefulness, adequacy, continued availability, or ownership. This product is copyrighted by the institution that created it. For more information or to get this report in an alternative format, call the Employment Security Department Labor Market Information Center at 360-507-9621. is an equal opportunity employer/program. Auxiliary aids and services are available upon request to individuals with disabilities. Language assistance services for limited English proficient individuals are available free of charge. Washington Relay Service: 711.
Contents Introduction... 1 Background... 1 Participation and performance... 2 Data validation review... 2 Design and method... 3 Initial analysis and visualization... 3 Tableau visualization... 3 Final analysis and visualization... 5 Findings... 7 Observations service date by data entry or edit date... 7 Observations service date by data entry date only... 8 Analysis... 10 Preliminary recommendations for how to improve data quality... 11 Glossary of terms... 13 June 28, 2018 Page i
June 28, 2018 Page ii
Washington state workforce data quality initiative An analysis of historical service data from SKIES: to inform data quality improvements Introduction Background Data quality is essential to determining the efficiency and effectiveness of any program or set of services. Given the importance of the workforce development services administered in state and local areas across the country under federal law, data quality becomes even more important. Not only does it jeopardize the ability to provide consistent, effective services to employers and job seekers, but errors can lead to U.S. Department of Labor (DOL) findings and even loss of grants and other funds. This historical data analysis takes a deeper dive into one-stop service data entry practices in Washington state, with a focus on Workforce Investment Act (WIA) Adult and Dislocated Worker (DW) services, and Trade Adjustment Assistance (TAA) services. This analysis establishes a foundation that Washington, and other states, may eventually expand into data quality monitoring tools for future data. Late entry of service data into a case management system has the potential of introducing inaccurate information. As the amount of time between when an interaction with a customer occurs and when staff enter the data into a case management system increases, the likelihood of inaccurate and incomplete data entry increases due to human error. There is a risk for any or all of the following: Staff simply forget to enter services. Staff forget exactly what happened and when, and make best guesses. Staff cannot find or misread handwritten notes on what occurred, and make best guesses. Staff infer what happened by looking at the electronic seeker record, and make best guesses. From 1998 through 2015, WIA established federal authorization, funding, guidance and performance measures for several key federal workforce programs. In Washington state, these programs are administered by the (ESD) and the 12 workforce development councils (WDCs), both in their previous WIA-authorized forms, and in their current, re-authorized forms under the Workforce Investment and Opportunity Act (WIOA). Many terms in this document may not be familiar to some readers. We have included a glossary at the end of this document, which may provide some insight. June 28, 2018 Page 1
Participation and performance Under both WIA and WIOA, program participation is service-based, where start and end dates correspond to service delivery dates. A participant s start date is the first date he or she receives a service, after a gap of at least 90 calendar days from any prior service. Likewise, a participant s exit date is the last day he or she received a service within the last 90 calendar days, except where the participant has been scheduled for future services. Once a participant exits, he or she falls into a performance cohort, a group of people DOL tracks to determine whether a state is meeting its outcome goals. The federal government has looked at several performance measures over time, including average or median wage, the number of persons working the second or fourth quarter after exit, and others. DOL also performs audits to ensure that states are entering data for the services they provide according to its rules and guidelines. Data validation review Following a 2013 DOL data validation review, ESD performed an internal audit of services entered into the Services, Knowledge and Information Exchange System (SKIES) case management system (CMS). Among other tests, the audit compared the service delivery date to the date it was entered or edited in the database. Several different patterns of data entry emerged. In some cases, service entry did not comply with Washington state policy requirements. Specifically, some services may have been entered or edited such that the participant would appear in an incorrect performance cohort. The WDQI grant offered Washington state the opportunity to further analyze these occurrences and develop recommendations for how to improve data quality in the future. June 28, 2018 Page 2
Design and method ESD looked to data visualization as an effective way to further explore and share case management data internally. Through an iterative process, ESD produced several data visualizations in R (an open-source statistical analysis program) and Tableau. After analysis using the Tableau tool, ESD determined that a final, slightly enhanced dataset would provide the most effective foundation for exploring the data. Initial analysis and visualization In consultation with the WorkSource System Performance team, ESD s Program Evaluation and Research Analysis (PERA) unit developed an R-based data visualization tool called SKIES Scan (named after the SKIES case management system) during the spring of 2017. The dataset extracted from the SKIES database consists of services created between April 2002 and April 2016, filtered for federally reportable services only and restricted to services associated with Adult, DW and TAA programs. SKIES Scan can be used to produce each of the following: 1. Service frequency distributions flagged for weekend/overload delivery: timeline frequency visualization of services delivered by individual staff within a single office. 2. Differences between maintenance dates and service dates by staff: tabulation of all staff within a single office by total services and by 7-day and 14-day backdates. 3. Differences between maintenance dates and service dates by service type: tabulation of all services within a single office by total services and by 7-day and 14-day backdates. 4. Differences between maintenance dates and service dates by program type: tabulation of services by program within a single office by total services and by 7-day and 14-day backdates. 5. Number of services by date and by office: scatterplot showing number of services by date and by office with services highlighted entered in a 20-day window around the end of the quarter (10 days prior to quarter end through 10 days after quarter end). 6. Numbers of unique individuals who had received at least one backdated service by office: tabulation of individual job seekers within a single office by total services, 7-day and 14-day backdates received. 7. Staff with high rates of late data entry: a variation on number 2 above. Tableau visualization Since data produced by the R application could not easily be shared with a larger internal audience, summary service and office-level tables were exported out of SKIES Scan and then combined into analysis sets suitable for import into Tableau. A single, interactive WDQI Report Tool was created to display the summary information for internal review. June 28, 2018 Page 3
This report tool is a five-page dashboard designed for easy navigation by an end user. Each page allows the user to view data by either seven or 14-day discrepancies between data entry date and service date. The pages are as follows: WA State displays state and workforce development area (WDA) summary data and a list of the top ten edited or backdated services for the entire state. WDA allows the end user to narrow the results down to a specific WDA. Summaries are provided for each of the offices within the WDA. Additionally, users can see a list of top ten edited or backdated services for the entire WDA. Office allows the end user to narrow the results down to an individual office. A top ten list of edited or backdated services for the office is also displayed. All services allows the end user to see all services provided across the state. The user can either see the complete list or compare results between multiple individual services. Definitions defines and characterizes the source data. Figure 1. Workforce data quality initiative report tool navigation Washington state SKIES services created between April 2002 and April 2016 Source: Washington State /WITS Within each of the report tabs, the user can select >7 days Back, or >14 days Back. The data within the report will change to reflect the choice made. Services are listed as they were in the source data. June 28, 2018 Page 4
Final analysis and visualization Based on visual analysis in the Tableau tool, ESD determined that additional data elements were necessary to complete an accurate review. Therefore an additional data pull was performed so that two parallel comparisons could be performed using the dataset: comparison of service date to date entered as well as to date edited. The new dataset enables statewide date difference comparisons by program and year. June 28, 2018 Page 5
June 28, 2018 Page 6
Findings Observations service date by data entry or edit date For the populations measured, 36 percent of all services statewide (585,794 of 1,628,391) were either edited or backdated seven or more days after services were provided and 33 percent of all services statewide (537,586 of 1,628,391) were edited or backdated 14 or more days after services were provided. Figure 2: All services statewide edited or backdated by program Washington state SKIES services created between April 2002 and April 2016 Source: Washington State /WITS Services edited or backdated more than 7 days Percent of services edited or backdated more than 7 days Services edited or backdated more than 14 days Percent of services edited or backdated more than 14 days Total services entered Program Trade Assistance Act 113,863 28.5% 104,321 26.1% 399,787 WIA Adult 215,372 41.1% 197,007 37.6% 524,361 WIA Dislocated Worker 256,559 36.4% 236,258 33.5% 704,243 Total 585,794 36.0% 537,586 33.0% 1,628,391 Edited or backdated services vary by program. We can break these numbers out by year. Figure 3: All services statewide edited or backdated by year Washington state SKIES services created between April 2002 and April 2016 Source: Washington State /WITS Percent of services edited or backdated 100% Percent more than 7 days 90% Percent more than 14 days 80% 70% 60% 50% 40% 30% 20% 10% 0% 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 A historical picture of edited or backdated services over time. June 28, 2018 Page 7
We can also break these numbers down by program. Figure 4: All services statewide edited or backdated by year, program Washington state SKIES services created between April 2002 and April 2016 Source: Washington State /WITS Percent of services edited or backdated 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% Trade Assistance Act more than 7 days Trade Assistance Act more than 14 days WIA Adult more than 7 days WIA Adult more than 14 days WIA Dislocated Worker more than 7 days WIA Dislocated Worker more than 14 days 0% 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 Edited or backdated services compared by program and over time could lead to program specific recommendations for data quality improvements. Observations service date by data entry date only The above figures capture differences between record edit dates and stated service dates. However, in some cases, staff are actually required to alter service records at later dates. This typically occurs when customers are involved in durational activities, such as occupational skills training, and staff need to provide an end date. Therefore, it is also helpful to look at deviations between initial data entry date and stated service date. For the populations measured, 23.1 percent of all services (375,950 of 1,628,391) were backdated seven or more days statewide and 18.9 percent of all services (307,151 of 1,628,391) were backdated 14 or more days. June 28, 2018 Page 8
Figure 5: All services statewide backdated by program Washington state SKIES services created between April 2002 and April 2016 Source: Washington State /WITS Services backdated more than 7 days Percent of services backdated more than 7 days Services backdated more than 14 days Percent of services backdated more than 14 days Total services entered Program Trade Assistance Act 74,736 18.7% 62,501 15.6% 399,787 WIA Adult 134,307 25.6% 106,757 20.4% 524,361 WIA Dislocated Worker 166,907 23.7% 137,893 19.6% 704,243 Total 375,950 23.1% 307,151 18.9% 1,628,391 Backdated service data as a subset of all edited service data. We can break these numbers out by year. Figure 6: All services statewide backdated by year Washington state SKIES services created between April 2002 and April 2016 Source: Washington State /WITS Percent of services backdated 100% Percent more than 7 days 90% Percent more than 14 days 80% 70% 60% 50% 40% 30% 20% 10% 0% 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 A historical picture of backdated services over time. We can also break these numbers down by program. June 28, 2018 Page 9
Figure 7: All services statewide backdated by year, program Washington state SKIES services created between April 2002 and April 2016 Source: Washington State /WITS Percent of services backdated 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% Trade Assistance Act more than 7 days Trade Assistance Act more than 14 days WIA Adult more than 7 days WIA Adult more than 14 days WIA Dislocated Worker more than 7 days WIA Dislocated Worker more than 14 days 0% 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 Backdated services compared by program and over time could lead to program specific recommendations for data quality improvements. Analysis In reviewing WDQI and other WorkSource data, ESD observed that there were issues in the SKIES CMS that may have resulted in differences between the dates services were provided and the dates they were entered. All services available in all situations in SKIES SKIES presented the same full list of services for any staff to enter at any time, regardless of an individual customer s circumstances. For example, the system allowed staff to enter adult-only services for a youth, or enter veteran-only services for a non-veteran. One specific example is the Reemployment and Eligibility Assessment (REA) program, which provided a specific set of services only for individuals receiving unemployment benefits in order to help shorten claim durations. Services were required to occur in order and consisted of an orientation with follow-up appointments. Despite this, staff were still able to enter follow-up services for non-claimants who did not receive an initial orientation. Service catalog not integrated into SKIES ESD maintained the state service catalog as a reference document outside of the CMS, and staff had to retrieve and consult it as necessary. Discussions with field staff have revealed that staff often entered services without consulting the service catalog to verify its proper use and without an understanding of the intent and limits of the service. Having the service catalog separate from the CMS certainly increased the likelihood that busy staff focused on serving customers would not review it every time they were entering services in the CMS. June 28, 2018 Page 10
Preliminary recommendations for how to improve data quality Washington state is currently hard at work implementing a new case management system. We anticipate that the efforts will lead to a more structured approach to data entry. Based on this analysis, ESD is recommending the new CMS incorporate the following: The CMS should limit the ability of staff to backdate services. If late data entry does need to occur, policy should provide guidelines for staff to follow to create a clear audit trail. Service catalog definitions should be available within the CMS. This will lead to more uniform understanding of service definitions across the state. Each service catalog revision should be reviewed by the system s stakeholders to clarify any potentially vague language. The new CMS should incorporate controls limiting the list of services available on a given form to those appropriate for the program enrollment of the customer being served. For example, if a customer is enrolled in a youth program, the list of available services should be limited to youth-related services. The new CMS should incorporate controls limiting the list of services available on a given form to those the staff person is authorized to deliver. For example, a WIA adult-funded staff person should not be able to deliver a trade service. If case notes are required to back up a service, a case note should be required by the CMS in order to save the service record. If there is a time limitation intended for a service, the CMS should include that limitation. June 28, 2018 Page 11
June 28, 2018 Page 12
Glossary of terms Adult: The workforce program under WIA or WIOA targeting job seekers who are 18 years of age or older. Case Management System (CMS): Software used by staff to enter data and track outcomes for customers in the WorkSource system. ESD:. Data visualization tool: Software that creates visualizations from underlying data in order to make information more accessible to an end user. Dislocated Worker program (DW): The workforce program under WIA or WIOA targeting job seekers who are unlikely to gain employment in their previous industry and need assistance in transitioning to a new career. Department of Labor (DOL): Federal department that provides oversight, guidance and funding to employment programs. Service catalog: The catalog of approved WorkSource services maintained by the ESD policy unit. The catalog defines each service, including whether and how it should count towards participation. Participant: A job seeker who has received at least one service that triggers or extends a participation episode. Participation episode: A time period specific to an individual participant. A job seeker becomes a participant on the date he or she first receives a significant service (after a gap of at least 90 calendar days from any prior significant service). That job seeker exits his or her participation on the date of his or her final service (prior to a gap of at least 90 calendar days from any subsequent services), as long as no future services are planned. Performance cohort: A group of people who have exited from a participation episode during the same quarter, and who are subsequently tracked by DOL to determine how individual states are progressing towards goals. Reemployment and Eligibility Assessment (REA): A federal grant funding specific services designed to shorten the duration of unemployment benefits claims. During fiscal year 2015, Washington state transitioned to providing services under the Reemployment Services and Eligibility Assessment (RESEA) grant. Services, Knowledge and Information Exchange System (SKIES): CMS used by Washington state workforce system. In May of 2016, SKIES was replaced with a new CMS. Trade Adjustment Assistance (TAA): A federal workforce program targeting employers threatened by international trade, as well as specific job seekers who have been laid off by those employers. Workforce Data Quality Initiative (WDQI): Grant given to Washington state to study data quality in the CMS. Workforce development area (WDA): Defined regional geographic areas in Washington state. June 28, 2018 Page 13
Workforce development council (WDC): Independent, non-governmental council that governs and manages a WDA. Workforce Investment Act (WIA): Was enacted on August 7, 1998, and was repealed and replaced by WIOA in 2014. Its intent was to move youth and people with significant barriers to employment into high-paying jobs. Workforce Innovation and Opportunity Act (WIOA): Was enacted on July 22, 2014, as a replacement for WIA. WIOA combines the authorization for several workforce programs, including WIA programs, into a single piece of legislation with an emphasis on streamlining and integrating programs and services. Wagner-Peyser (WP): Act of 1933 that established a nationwide system of public employment offices. June 28, 2018 Page 14