PROMISE: Lessons learned from six model demonstration projects through the Promoting Readiness of Minors in Supplemental Security Income Project

The overriding evaluation goal was to document and describe the experiences of the PROMISE Model Demonstration Projects (MDPs) leadership in implementing the planned activities of these projects. Recording and capturing these experiences from the perspectives of those engaged in leadership roles is of critical importance to federal and state agencies and policymakers in building on these efforts in the future. Specifically, we wanted to develop an understanding of the implementation challenges and successes experienced by the MDPs during various stages of project implementation over the five-year federal funding period. This information, in turn, was used to identify specific lessons learned that were experienced by the six MDPs. The evaluation approach was sensitive to the understanding that each MDP was unique and approached project implementation with differing strategies and activities. It was important to capture the individuality as well as commonalities across the six MDPs in forming partnerships and implementing the required services. This evaluation summary sought to address the following questions —

• What do you view as the most important strategies that supported partnership development and the implementation of project services?
• With regard to each strategy, what were the supports and constraints that influenced implementation by your project?
• In assessing what your project accomplished over the five-year federal funding period, what will be translated into sustainable changes in service delivery policies and practices to support SSI youth and families?
• What do you view as the most significant lessons learned over the five years of project operation?

Data Sources

There were three primary data sources used to obtain information in relation to these questions: Annual Project Directors’ meeting notes, document reviews, and a project directors’ focus group discussion. A content analysis method was used to identify major themes from the various sources of information available.

Annual Project Directors’ Meeting Summary

The PROMISE MDP project directors participated in an annual two-day meeting in Washington DC every year of the project period, 2013-2019. These annual meeting discussions were facilitated by the PROMISE Technical Assistance (TA) Center and OSEP project officers. Meetings focused on specific implementation activities and actions taken by the MDPs, modifications and changes in implementation plans, policy and system-level challenges experienced during implementation, project information dissemination strategies, successful progress and accomplishments experienced, and state sustainability strategies and plans. These meeting discussion summaries were reviewed and basic themes identified.

Document Reviews

An extensive review of over 80 documents was conducted by the PROMISE TA Center staff over the period 2014-2018. These documents included —

• Technical Assistance Plans. All six PROMISE project directors received ongoing technical assistance from the PROMISE TA Center. Technical assistance plans were developed with each of the MDPs and reviewed annually. Technical assistance included meeting facilitation and consultation with individual state MDPs and ongoing consultation and reporting with the OSEP project officer. These plans identified specific needs for support, strategic goals and priorities, and planned implementation strategies for PROMISE TA Center support. The TA plans served as a useful chronological documentation of each project’s planning and implementation experiences.
• OSEP 3+2 Reviews. Comprehensive, individual project reviews were conducted with each MDP midway through the five years of federal funding. Each MDP developed a “briefing book” that contained detailed information about project activities, progress related to performance measures, project evaluation methods and strategies, initial lessons learned, ongoing challenges and potential solutions to address these challenges, and proposed activities focused on the project’s future implementation. Each of the PROMISE MDP directors and other leadership staff presented this information to a panel of federal partners for review and comment as a means of assessing progress. Following the OSEP 3+2 Review, summary reports, including reviewer comments and OSEP recommendations for each of the MDPs, were prepared.
• Annual Performance Reports (APRs). In accordance with federal requirements, the MDPs each submitted an APR to OSEP. The APRs included information on project progress toward identified goals, performance measures, and project accomplishments to date. These reports served as a useful source of information in gauging individual project progress across the five years of implementation.
• Social Security Administration (SSA) Request for Information (RFI). SSA released a request information in January 2018 (Docket No. SSA-2017-0049).  This request RFI sought public input on strategies for improving the adult economic outcomes of youth ages 14 to 25 with disabilities receiving SSI. MDPs were not required to submit a response to the RFI, however, several of the MDPs and the PROMISE TA Center provided written responses to SSA for the RFI. The public comments submitted to SSA from the MDPs were included in this analysis as an additional, supplemental source of information describing project implementation progress and recommendations to improve youth post-school outcomes.
• Mathematica Policy Research (MPR) MDP Reports. MPR, funded through a contract with SSA, served as the national evaluator for the PROMISE project, conducting a number of reviews and assessments with each state MDP over the five years of the project. More specifically, MPR was responsible for collecting and analyzing all data from the youth intervention and control groups as part of the randomized control research design for the PROMISE program. In 2018-2019, MPR released interim impact and process reports for each of the six MDPs. These provided a comprehensive overview of each project’s activities, challenges, accomplishments, and lessons learned. Information from these reports was also used in this analysis.

Project Directors’ Focus Group

At the July 2018 OSEP Project Directors’ Meeting, a focus group was conducted with the MDP directors and staff to identify and document responses to a series of questions designed to probe, understand, and document various aspects of program implementation and lessons learned. Specific questions posed to the MDP project directors included —

• Did your state’s PROMISE project accomplish what you envisioned it would, and if not, why not?
• What factors helped to make it a success, and what is the value of what has been achieved by your project?
• What has been the impact of your PROMISE project in the communities in which it has engaged?
• What would you have done differently than originally planned to overcome implementation challenges?
• Of what you accomplished, what will be translated into systems changes, and how do you hope to affect those changes at the state level, going forward?
• The focus group was facilitated by PROMISE TA Center staff, with a summary report developed documenting responses to these questions of interest.

Conclusion

In this section, we’ve summarized our data sources and analysis strategies in seeking an understanding of the challenges and successes experienced by the MDPs during the five-year project. Through that analysis, we identified many, but certainly not all, of the major collective lessons learned through the experience of the MDPs. Twelve of these lessons are presented and described in Section IV of this report. They are lessons that can potentially assist other federal, state, and local efforts in supporting SSI youth and their families to achieve positive outcomes in education, employment, and reduced dependency on SSI.