UWSEM SIF: IMPLEMENTATION EVALUATION INTERIM REPORT - OCTOBER 16, 2015

Transcription

1 UWSEM SIF: IMPLEMENTATION EVALUATION INTERIM REPORT - OCTOBER 16, 2015 PREPARED FOR: Jeffrey Miles Social Innovation Fund Manager United Way for Southeastern Michigan Jeff.Miles@liveunitedsem.org SUBMITTED BY: Implementation Evaluation Team Child Trends scaal@childtrends.org

2 TABLE OF CONTENTS Introduction Goals of the Implementation Evaluation... 3 Method Data Collection and Information Sources Sample... 4 Program facilitators... 4 SIF Leads... 4 Internal Data Staff... 4 Evaluators... 4 Staff at UWSEM Recruitment Data Collection and Analysis... 5 Findings Program Implementation... 5 Participant implementation... 6 Participant recruitment and retention: Subgrantees are making progress despite the hurdles... 6 Program staff changes and capacity Program Evaluation Activities... 7 Improved strategies for data collection Partnership with Community Organizations... 7 Partnerships with external organizations... 7 Partnerships within the SIF portfolio Initiative-Related Perceptions... 7 Finding match funds Common Outcomes and Indicators (COCI) Framework... 9 Using COCI to communicate results... 9 The COCI provides a perspective on their collective impact... 9 Fitting square pegs into round holes... 9 Recommendations and lessons learned for the COCI Summary Recommendations for the UWSEM-SIF Initiatives

3 INTRODUCTION For the past two years Child Trends has been working with United Way for Southeastern Michigan (UWSEM) to conduct a portfolio evaluation of their Social Innovation Fund (SIF) project. The goal of the project is to prepare young children in the Detroit region for success in school, active citizenship, productive work, and healthy and safe lives. The portfolio evaluation includes two main components: an outcomes evaluation, and an implementation evaluation across participating subgrantees. To date, Child Trends has completed two reports on the implementation; an initial report, disseminated in November 2014, summarized baseline findings from the first round of program site visits Child Trends conducted in the spring of 2014; a second report, disseminated in February 2015, summarized findings from our second round of visits in fall This report summarizes key findings from the third round of data collection, in the spring of 2015, and focuses on progress made by subgrantees and UWSEM to that date. Between the first and third rounds of data collection, four organizations made an early departure from the UWSEM-SIF portfolio; thus, in this round Child Trends conducted phone interviews with seven current subgrantees, and one subgrantee preparing for an early departure. The current report delves more deeply into the successes the initiative as a whole experienced since our previous site visit, as well as identifying new and recurrent challenges. Additionally, we present the subgrantees and UWSEM s perceptions of the progress made on program implementation, their Subgrantee Evaluation Plans (SEPs), and their perceptions of the work done around the Common Outcomes and Indicators Framework. This report also highlights participants reactions to our findings from the previous round of data collection on the implementation. Goals of the Implementation Evaluation There are three general goals for the portfolio evaluation, of which the implementation evaluation is a major component: (1) to assess the progress each of the organizations has made towards their individual program and SIF goals; (2) to obtain information on the progress of the UWSEM initiative, as a whole, in achieving the SIF goals; and (3) to determine how organizations are working together toward common outcomes. Documenting the organizations progress, challenges, successes, and their collective effort to reach common outcomes, will help identify cross-site themes that may be useful for the UWSEM-SIF initiative, the SIF as a whole, and the early childhood field at large. The following research questions guided the third round of implementation data collection: Research Questions: 1. What is the progress each organization continues to make in its program, in the SIF initiative, and in its SEP? 2. What barriers persist, and what has been successful in overcoming them? 3. What has facilitated subgrantees successful implementation of their program and SEP? 4. What is the level of success, to date, that the UWSEM initiative demonstrates toward achieving SIF goals? METHOD Data Collection and Information Sources A series of phone interviews were conducted. UWSEM and Child Trends agreed that, since rapport had been established between Child Trends and subgrantees in previous face-to-face interviews, it made sense to conduct phone interviews for this round of data collection. This option would also be most time- and cost-efficient. For this report, the Child Trends research team summarized interview data from key informants in seven subgrantee organizations in the UWSEM-SIF initiative, and one early-departure subgrantee. These organizations serve participants in a variety of settings, including their homes, early childhood centers or preschools, elementary schools, and the community at large. While each program focuses on improving school readiness among children in southeast Michigan, the target populations programs serve can include parents, children, or teachers. Recruitment and data collection procedures are described below. Data were also collected from phone interviews with UWSEM staff. perceptions of the work done around the Common Outcomes and Indicators Framework. This report also highlights participants reactions to our findings from the previous round of data collection on the implementation. 3

4 Table 1 Overview of Subgrantees SIF Programs. Subgrantee ACCESS Detroit Parent Network (DPN) Living Arts Macomb Intermediate School District (MISD) National Kidney Foundation of Michigan (NKFM) Southwest Counseling Solutions (SWSOL) Starfish Family Services The Guidance Center Program Overview The ACCESS school readiness parenting program is made up of four main components: ESL programming, parenting education and skill building, parent and child literacy activities, and intensive case management. Pathways to Literacy is a project in which parent-coaches provide teaching strategies, support, and modeling, focused on children s language, cognitive and communication skills. The El Arte Early Learning project increases children s exposure to the arts by providing direct services in Head Start classrooms, and by training teachers to engage children in dance, music, and drama activities that boost literacy and other kindergarten readiness skills. A collaborative project of MISD, Macomb Family Services, and Leaps and Bounds Family Services, the Ready Schools, Ready Communities program prepares children for kindergarten through an integrated model that includes a literacy-based outreach and engagement approach, home visitation using the Parents as Teachers curriculum, and community-school collaboration to ease the transition to kindergarten. Project for Early Childhood Health (PEACH) aims to improve health outcomes of children through three integrated strategies: delivery of the Regie s Rainbow Adventure, individualized coaching with parents and caregivers, and assisting early childhood settings by supporting their adoption of policies, procedures, and practices that promote health. The English Language Learners Program uses a family literacy model to improve literacy among young children from non-english-speaking households. The Inkster Family Literacy Movement aims to prepare children for school and teach parents about literacy through a mix of community campaigns to create a literacy-promoting culture within Inkster, providing fun and accessible family literacy programs, services and events. The Family/Community Engagement Initiative aims to build social competence and emotional maturity through parent engagement, by implementing the evidence-based Families and Schools Together program array, and the Community Organizing and Family Issues model. Sample A total of 49 participants took part in phone interviews in May and June of 2015 (see Table 2). To gather information on program operations, day-to-day activities of front-line staff, and information pertaining to external evaluation activities, Child Trends project staff conducted individual and group interviews with four key stakeholder groups: (1) program facilitators, (2) SIF leads, (3) programs data staff, and (4) external evaluators. In the case of some organizations, more than one of these roles is assumed by the same individual, who wears many hats. For example, the staff member serving as SIF lead may also serve as the internal data manager. In one case, the current SIF lead and former SIF lead for the organization were both interviewed. Program facilitators: Thirteen program facilitators participated in interviews. In some situations, facilitators participated in interviews as a group, due to limited time availability. Facilitators were a mix of part-time and full-time staff members, depending on program staffing structure, but all worked directly with clients to deliver the program. Three facilitators had not been interviewed previously for the project, because they were either not available or were new to the organization. The other ten facilitators had been interviewed in 2014 and remained in similar work roles in the fall. SIF Leads: Twelve respondents were interviewed as SIF leads. Ten were currently the primary managers of the SIF grant for their program; two respondents had formerly been the SIF leads for their organization, but recently had transitioned or shifted responsibility to a new person. However, they joined the call to provide additional information that the new SIF lead could not provide. All but one respondents had been interviewed in October 2014; one SIF lead had been previously interviewed as a facilitator. These two respondents had transitioned to the SIF lead role since the last site visit. Internal Data Staff: Ten respondents were interviewed during the internal data manager interview. Two of these people served additional roles, such as SIF lead. Two respondents had not participated in a previous interview for this project, because they were new to their respective organizations. Evaluators: Twelve external evaluators were interviewed. At four organizations, two individuals were present for this interview. All evaluators or evaluation teams had been interviewed in October There was only one new respondents from one evaluation team. Staff at UWSEM: Five UWSEM staff were interviewed. These included two new staff members who were hired since October Four staff members work exclusively on the SIF at UWSEM, while one has responsibilities in the SIF initiative and other early childhood work at UWSEM. 4

5 Table 2 Number of Interviewed Respondents and their Role in the UWSEM-SIF Initiative Role Number Interviewed SIF Lead 12* Program Facilitator 13 Internal Data Staff 10** External Evaluator 12 UWSEM Staff 5 *This number includes ten current SIF leads and two past SIF leads. **Two internal data staff simultaneously served as SIF lead. Recruitment To recruit participants, UWSEM staff contacted subgrantees to link them with Child Trends researchers. Subsequently, staff at Child Trends corresponded with subgrantees SIF lead and UWSEM-SIF staff to coordinate phone interviews. SIF leads at each subgrantee organization were asked to identify and invite program facilitators, internal data managers, and external evaluators to be interviewed. SIF leads at each organization helped to coordinate interview times for staff. Data Collection Procedures The Child Trends research team conducted individual semi-structured interviews and, when needed, group interviews with program facilitators. Group interviews were conducted when program facilitators were only available at one time. Individual and group interviews were approximately one hour long; all interviews were voluntary and audio-recorded with verbal, informed consent from participants. A note-taker documented responses to questions during the interview. Individual participants who had not been interviewed previously for this project (n = 9) were asked to provide a description of their role in the organization with regard to the SIF initiative. Respondents who had participated in Child Trends previous site visit in April, 2014 (n = 40) were asked about any changes to their role or responsibilities, and about the current progress of SIF work at their organizations. The interview protocol focused on any changes to goals, program implementation, data collection and management, or use of funds, which had occurred since the last site visit. All respondents were asked about current successes or challenges, and their highest priorities for the coming year, pertaining to the SIF initiative. Program facilitators and SIF leads were asked to describe any recent changes to program structure, setting, recruitment, staff, goals, or community partnerships, as well as changes to the evaluation design, and to describe current progress relating to data collection and management. SIF leads were also asked to comment on Child Trends report of findings, as of the last site visit, regarding the biggest successes and challenges of the initiative. Internal data managers and external evaluators were asked to describe any changes to the evaluation design or measures and collection methods. External evaluators were asked about the current progress of the evaluation plan, and to describe analyses and use of data thus far, as well as any planned for the future. FINDINGS In this report, we highlight the key findings from the third round of data collection, and provide an overview of progress made by the organizations and UWSEM since the beginning of the initiative. During this round, Child Trends generally observed continued progress across all sites. All seven current subgrantees were fully implementing programming and collecting data, and three were getting ready to start analyzing data. Participants were still encountering challenges similar to those we observed at our initial site visit, such as raising match funding, and undertaking a rigorous evaluation. While some subgrantees had improved their recruitment and retention, others still faced challenges in this area. Program Implementation In our last report, we reported that participant retention, compliance with evaluation activities in their SEPs, and raising their match funding continued to be challenges for subgrantees. While these challenges were echoed during this round of data collection, some subgrantees reported using different and creative approaches to help overcome them. Some subgrantees reported improvements in data capacity, community partnerships with other SIF subgrantees, and plans to scale up their program. The section below highlights these successes and recurring challenges. 5

6 Program Implementation Most programs were running as planned, with minor tweaks having been made to content or implementation schedules. As we observed in October of 2014, these changes were made to tailor programing to the cultural context of the participants, including offering programming at more convenient times or days, offering the program in Spanish, adding more parent-child literacy opportunities, offering transportation, or coaching staff on components of the program where improved performance was needed. Programs made alterations in implementation during the cycle primarily based on anecdotal evidence, or by analyzing recruitment and retention numbers. One subgrantee, however, identified areas where staff required coaching, based on focus-group findings held by their evaluator and subsequent discussions. Participant recruitment and retention: Subgrantees are making progress despite the hurdles. During previous years, most subgrantees reported that recruitment and retention were two of the primary challenges in implementing their SIF-funded program. While there have been some successes in participant recruitment, retention challenges persist across the majority of organizations. Participant recruitment: Only one subgrantee had met their recruitment goal for the year at the time of these interviews. Of the balance, half felt they were on track to meet the goal by the end of this program cycle, but the other half expressed concern over their ability to reach the goal by the end of the cycle. The organizations that expected to meet their goal, along with the organization that had met its goal, used different strategies to recruit participants. The most common and successful recruitment strategy cited was enlisting the help of other professionals working in the communities. More specifically, these organizations reported investing more time in establishing trust with teachers, and giving teachers a bigger role in the recruitment process. According to the facilitators from these organizations, teachers subsequently became more supportive of the program, and pivotal in ensuring surveys were completed and returned on time. One of these four organizations noted this was the first time it involved teachers, and as a facilitator noted: [A new recruitment strategy is that] I reached out to the teachers. Now, we re in our third year. A majority of the teachers who participated in the program are really on board with it, and they have been great resources. That really worked for me, and I hadn t done that in the last few years. Additionally, two organizations on track for meeting their recruitment goal reported they increased the school presence of program staff to recruit participants and reduce the burden on teachers; while this was deemed a successful strategy, the drawback was that it required a big commitment of staff time, as one facilitator noted: The recruitment has gotten better with the [community wide events], but it takes a lot of staff time. I think we have gotten better results and have better relationships with our sites by having our staff go there and be a physical presence and trying to take some burden off of the teachers. Other common recruitment strategies that most subgrantees cited included word of mouth, visiting libraries, churches, radio stations, and adding Spanish-language programming or families. Organizations that expressed concern over their ability to reach their goal cited recruitment challenges similar to those reported in previous site visits. The most common challenge was that of zip code restrictions on recruitment, and this continues to be challenging for the majority of organizations. Some reported that, despite having participants interested in their program, they were not able to count them towards their recruitment goal, as one SIF lead notes: For recruitment, it is difficult because our agency serves a much larger service area than the SIF zip codes that were identified. So, recruiting people in the specific zip codes is difficult, we have a lot of people from outside, those who want to join, and we would easily meet our numbers if they could be counted Additionally, some reported that they reached saturation in recruiting families from their respective zip code, and they were not able to recruit outside of this zip code. 6

7 Participant retention: The vast majority of subgrantees mentioned that participant retention remained a challenge, even when they had been able to meet their recruitment goals. Subgrantees targeting parents seemed to have the most challenges in retaining participants. As reported in previous site visits, the main reasons parents stop attending the program included family responsibilities, conflicts with work, and lack of transportation. To remove some of the barriers parents face in attending the programs, subgrantees found various solutions. For example, two organizations offered transportation and noted that the strategy was successful. Unfortunately, due to lack of funds and availability of a vehicle, these efforts could not be sustained for the long term. Program staff changes and capacity All current subgrantees had experienced some type of internal staffing change since the last interview, whether turnover or expansion. The majority of organizations had a SIF staff member leave since the last visit; most were program facilitators or internal data staff. In general, these changes did not seem to create unmanageable challenges for organizations, and were regarded as typical and not as a result of the SIF initiative. During our previous site visit, subgrantees reported having to hire new staff because programs were expanding, due to the funds acquired as part of the initiative. Similarly, during this round of data collection, a few subgrantees reported hiring, or being in the process of hiring, new program facilitators to accommodate program expansion, and to be able to better serve their target population in culturally sensitive ways. For example, two SIF leads reported that, although they were fully staffed, they lacked a staff member who spoke the language of the families they serve, and they felt limited in their ability to serve or collect data from these families. Program staff changes and capacity All current subgrantees had experienced some type of internal staffing change since the last interview, whether turnover or expansion. The majority of organizations had a SIF staff member leave since the last visit; most were program facilitators or internal data staff. In general, these changes did not seem to create unmanageable challenges for organizations, and were regarded as typical and not as a result of the SIF initiative. During our previous site visit, subgrantees reported having to hire new staff because programs were expanding, due to the funds acquired as part of the initiative. Similarly, during this round of data collection, a few subgrantees reported hiring, or being in the process of hiring, new program facilitators to accommodate program expansion, and to be able to better serve their target population in culturally sensitive ways. For example, two SIF leads reported that, although they were fully staffed, they lacked a staff member who spoke the language of the families they serve, and they felt limited in their ability to serve or collect data from these families. Program Evaluation Activities Improved strategies for data collection In our last report, the common theme among subgrantees was that they had improved their capacity to collect data and had adopted more rigorous data collection methods as a result of taking part in the UWSEM-SIF initiative. This sentiment was echoed during this round of data collection, and the majority reported feeling more comfortable with the methods they had instituted and the overall process. This was in part attributed to program staff having a better understanding of the importance of data and knowing how to collect it, as well as having well-functioning processes, as an internal data person noted: We re in a better place. We have our established system in place. Last year was a building year, we created new surveys, new databases, and figured out when things needed to be done. We have a database, protocols, and surveys all in place. In addition to feeling more comfortable with data collection, subgrantees also made adjustments to improve participant response rates. For instance, one organization hired a bilingual staff member who could help participants complete surveys in their native language, and another organization reported investing more in incentives to encourage participants to complete surveys and other assessments, as one internal evaluator noted: We invested more in incentives. That s a concrete thing we did. It has helped a lot Another grantee was able to offer participants the option to fill out their survey online to increase the number of completed surveys. Overall, participants seemed to have reached a point where they understood the challenges involved with implementing a rigorous evaluation, and had adopted methods to overcome those challenges. 7

8 Partnerships with Community Organizations While all subgrantees came to the UWSEM-SIF initiative with some partnerships in place, the majority had strengthened these or developed new ones since they joined the initiative. However, few subgrantees reported developing new partnerships with organizations within the UWSEM-SIF portfolio. Partnerships with external organizations Conversations with SIF leads and program staff indicated that subgrantees had continued their community partnerships to help implement their SIF-funded program and/or to recruit participants. For example, subgrantees implementing a schoolbased program model continued their partnerships with schools or Head Start programs, and in a few situations these partnerships had strengthened or grown. For example, two subgrantees were operating at new sites or in new regions since the last site visit. Another example of how partnerships grew as a result of the UWSEM-SIF initiative came from a program facilitator who recounted how their partnership with one school had grown and strengthened their relationship with the wider organization of Detroit Public Schools: there is more collaboration in the community, especially between schools and community agencies. in the past, schools were pretty isolated from community organizations, we ve really been able to bring that together to offer supports to schools. I think that is huge. [SIF] has allowed us to partner with other organizations in the region who are doing [similar work]. As stated above, most organizations continued their community partnerships for purposes of participant recruitment. Partnerships included libraries, churches, and schools. One subgrantee spoke about how their partnership with libraries, while not new to this program cycle, was successful as a way of making their program and resources available to more families. Interestingly, the vast majority of subgrantees shared the sentiment that partnerships were a strategic investment for program sustainability and, thus, were vital for their organization, even after the initiative comes to an end. Partnerships were needed for program implementation, participant recruitment, and for referrals to better serve the community. Partnerships within the SIF portfolio In terms of partnerships within the SIF portfolio, few subgrantees worked together, and no new partnership activities were reported among subgrantees. As reported previously, the vast majority of subgrantees saw the competitive recruitment and funding environment as barriers to their forming such partnerships. That is, subgrantees felt they were competing with each other to fulfill their recruitment goals and to raise funds. UWSEM staff anticipated more opportunities for partnership as subgrantees create sustainability plans, and they saw the newly hired UWSEM staff member as a way to facilitate these cross-portfolio relationships. Initiative-Related Perceptions Finding match funds In terms of match funding, the common theme was positive, as most subgrantees had met their match for the year, or felt they would be able to meet the match by the deadline. Those who had already met their match were doing so through a combination of grants and their own organizational funds. Roughly half of subgrantees had acquired new funding this year, in the form of small grants. Although most subgrantees had raised their match funding or were hopeful that they would do so, most SIF leads agreed that this was still one of the most challenging parts of the initiative. The same challenges that they had faced throughout the initiative persisted, including steep competition in the region for early childhood funds, competition with UWSEM, and the level of funding they were required to match, as one SIF lead noted: It s hard to raise that much match when you re a community based organization. It was easier for [us] because we are a large organization, but it was still difficult to put all of our organizational resources into one program when we have [lots of] programs. The struggle to raise the necessary funds also created uncertainty around being able to provide programming. One subgrantee expressed the stress of planning programming without knowing what funding will be available: 8

9 The struggle to raise the necessary funds also created uncertainty around being able to provide programming. One subgrantee expressed the stress of planning programming without knowing what funding will be available: It s challenging. It s a lot of money to match. We re fortunate to get the bulk of the match through [a funder] and then smaller grants from other entities. We don t know if we ll have [that funding] next year. [ ] That s always very stressful. What are the essential parts of our programming we want to sustain if funding is unavailable? How would we have to downsize? Change the program? The greatest need that we have is finding funds. A minority of SIF leads felt that if UWSEM were to help subgrantees raise their match, then this would not be as great a challenge for the whole portfolio, since the United Way has national recognition and is able to attract more high-profile funders. Interestingly, during this round of interviews, UWSEM staff reported that raising their own match was also an ongoing challenge. According to one UWSEM-SIF staff member, the challenge was in part due to the match requirement being too great to be supported by individual donors year after year. Thus, according to this staff member, one solution would be to have intermediaries come to the SIF initiative having major donors already lined up. Common Outcomes and Indicators (COCI) Framework SIF leads, evaluators, and UWSEM staff were asked about the Common Outcomes and Indicators (COCI) framework, including its value for the overall initiative and for their own organizations. They were also asked for feedback on the process of defining the COCI framework and using it for aggregating the data, and whether participation in this aspect was burdensome or had any drawbacks. The following describes some of the most common themes that emerged regarding the COCI work. Using the COCI to communicate results Subgrantees understood that the purpose of the COCI was to demonstrate the collective impact the UWSEM-SIF has made in getting young children in the Detroit region ready for school. There was common sentiment among the vast majority of SIF leads, facilitators, and internal and external evaluators that this was important in order to communicate the results of their collective work in a way that made sense to others. For example, one SIF lead thought the document the UWSEM created was helpful, as it showed the common outcomes and indicators in a clear way without jargon. The majority thought that the COCI would be especially important to communicate to funders, both recurring and new, about their program s contribution to the collective impact that the initiative has accomplished The COCI provides a perspective on their collective impact The majority of SIF leads and program facilitators, and a minority of evaluators, perceived the COCI work as a motivator for proceeding in their work, despite the great efforts required. These respondents saw the Common Outcomes and Indicators framework as a concrete way to see how their individual work could have a collective impact in young children in the Detroit region, their ultimate mission. Fitting square pegs into round holes The majority of subgrantees agreed that the process for coming up with the Common Outcomes and Indicators framework was challenging, but necessary. A minority of SIF leads and evaluators explained that this was a challenging process because of widely varying programming and targeted child outcomes across the portfolio. One evaluator thought the COCI work was analogous to fitting square pegs into round holes, since, although all subgrantees are targeting early childhood, their implementations are very different. In addition to perceiving the COCI work as challenging, the majority of SIF leads, evaluators, and program facilitators were concerned about the way data would be pooled and conclusions drawn. Some were concerned that their data be presented accurately, and the reasons for this concern varied. For example, one SIF lead and evaluator stated that their school readiness outcomes were exploratory and their inclusion in the COCI work was concerning. Others had concerns regarding how their measures were being used in this work, especially when their measures had not been validated with their population. Another concern expressed was that the COCI may be limited in the meaningful findings it is able to produce. A minority of respondents across subgrantees and evaluators were also concerned that the number of organizations contributing to each outcome was inconsistent, and that this issue would be amplified now that the portfolio had lost subgrantees. Thus, these respondents questioned the COCI s ability to show a valid collective impact, as one informant shared: It s a little tricky and we re wasting our time being unrealistic about how strong of a case we re making that we re a part of one project. 9

10 In spite of these concerns, subgrantees reported that being a part of the COCI was not burdensome and, at this point, had no drawbacks. One subgrantee shared this sentiment by saying, We just get to sit back and see how it goes. We re not involved in any of the headache. A minority of subgrantees candidly stated that at first it seemed that there may be burdens in having to add a common measure for the COCI work, but that UWSEM s expectations were flexible and accommodated what worked best for organizations, and subgrantees were thankful for that. UWSEM-SIF staff s perception of the COCI work was that the COCI could be moved further along and at a faster pace. They acknowledged that this would depend on their staff to facilitate, but noted that they would appreciate more guidance from Child Trends to push the work forward, such as suggesting deadlines for what UWSEM needs to produce, by when. Recommendations and lessons learned for the COCI Almost all subgrantees said that the COCI process and associated decisions should have happened much earlier in the life of the initiative. Some suggested that common outcomes could have been a part of the initial proposal or grant-making stage, ensuring that all were on board. The vast majority remarked that it was challenging and surprising that the process took place after their SEPs had been approved. These subgrantees agreed that if the COCI had been established early in the initiative, it would have allowed for more alignment in outcomes and measures, would have helped them to develop their SEPs, and could have been used to communicate to funders and about the initiative in the region. As one evaluator said, Not having [COCI] done early not only was difficult and inconvenient, but also limited our collective impact because by the time they started doing it, all the agencies had already chosen assessments, gotten IRB approval for when they were going to give these assessments and collect this data. Had it been done earlier we could have come up with a larger more collective strategy. Aside from starting the process earlier, another lesson learned from grantees for future endeavors is to establish data sharing or use agreements and other necessary infrastructure to enable data to be shared and aggregated at the beginning of the process. SUMMARY SIF subgrantees continued to make progress implementing their program and their SEPs. Despite the continued challenges, subgrantees used what they have learned in the past two years to their advantage, and had developed strategies to address challenges early on. Their commitment to the community and passion to serve participants continued to be the driving force behind their efforts. Subgrantees continued to face barriers around participant recruitment and retention, but made strides in their data collection process, though there was still room for improvement. Participants also became more strategic when maintaining existing partnerships and creating new ones, ensuring these relationships were beneficial for both parties and could become a key factor in the sustainability of the program after the initiative comes to an end. Most subgrantees agreed that, while raising funds continued to be a challenge, they would meet their match for the year. Subgrantees agreed that their participation in this initiative was valuable in improving and expanding their data capacity, and, with the support of UWSEM, had also contributed to their plans for scaling up. In general, the Common Outcomes and Indicators work was perceived as valuable, although there were concerns that it may not be a valid reflection of subgrantees programming and logic models. Recommendations for the UWSEM-SIF Initiative Findings from this round of interviews inform our recommendations for the UWSEM-SIF initiative. We continued to see trends similar to those we observed during the previous data collection, including barriers related to participant recruitment and retention, evaluation activities, and raising match funds. In addition to the recommendations we have made in previous reports, this section brings attention to recommendations around 1) building partnerships, and 2) investing in program sustainability. 10

11 Strategic partnerships go a long way. Subgrantees noted that one of the strategies used successfully to alleviate the burden of data collection and participant recruitment and retention was to become more involved with their external community partners. It is important that organizations understand the value of these relationships early on in the initiative, so they can begin addressing these challenges. Being able to communicate program goals and the purpose of the evaluation helped them get buy-in from schools and school staff. Intermediaries implementing a SIF initiative may want to support and encourage subgrantees in developing messaging strategies and in taking advantage of partnerships to facilitate implementation during and after the initiative. Facilitate and encourage networks among subgrantees. While some subgrantees reported establishing new partnerships in the community and continuing to nurture old ones, none seemed to have invested time in creating partnerships with other subgrantees within the SIF portfolio. A number of subgrantees believed this was a result of competition for funding sources. UWSEM might consider spending time working with subgrantees on ways they could partner or serve as a resource for each other throughout and after the initiative. UWSEM has created a new position that will facilitate this conversation among organizations, and aid them in establishing a network; perhaps new partnerships will emerge by the next round of data collection. Planning for sustainability. A small minority of subgrantees voiced uncertainty about the continuation of implementation once the initiative comes to an end; one subgrantee mentioned they had begun brainstorming their plan to expand their programming. As the initiative comes to an end, it is imperative for subgrantees to have the tools necessary to sustain their efforts to continue implementation. UWSEM plans to institute a new staff role that will be entirely dedicated to provide guidance to subgrantees on how to plan for scaling-up and replicating their programs. 11