Outcome Measurement Program Case Studies

Framework Domain: Intervention Characteristics

External vs. Internal Development:  The first factor considered is the degree to which stakeholders “buy” into the tool. CFIR research indicates that those programs that are not developed by the stakeholders as a “grassroots” solution to a problem or challenge and are centrally controlled, risk implementation failure.  The IM4Q avoids this challenge as it is an internally developed measurement that grew out of the state’s efforts to improve quality monitoring and assurance.

As more people are living in individualized settings, the ability to recruit people becomes more difficult.  While those within the system may understand the value of IM4Q to improve quality, end users of services and their families do not necessarily see the value for their own circumstances.  Increased efforts to communicate how participation may benefit individuals may overcome this challenge, for example, by explaining how IM4Q may address service quality issues at both the service and system level.  Because the IM4Q is implemented at the local level, there is an opportunity for local organizations who understand the community context to communicate the IM4Q’s purpose and utility to their clients and communities.

Evidence Strength and Quality: The CFIR foundation literature indicates that a tool’s success is strengthened when stakeholders believe in the tool’s credibility.  Perceptions of tool relevance and effectiveness may derive from a range of sources such as the influence and views of colleagues, studies of reliability and validity, as well as pilot results and feedback from implementation teams. 

The EDE is based on the NCI-IPS survey.  The NCI has undergone some reliability and validity studies and work continues in this area. However, reliability and validity evidence of the EDE as a complete instrument is very limited.  For instance, one study reported high the inter-rater reliability scores (74 out of 75 questions had 85% or higher agreement; Goreczny et al., 2015). Summative and formative evaluations of the Considerations process have not been completed.

Relative Advantage:  The CFIR research evidence supporting this construct finds that there is stronger implementation when stakeholders view their selected intervention as providing an advantage over similar interventions.

The most significant advantage of the IM4Q is its independent monitoring process that limits conflict of interest with ODP, the counties, or providers when decisions are made. The results may provide a more critical view of how the system is performing due to this independence. . In addition, it reflects personal voices and unique aspects in Pennsylvania that other programs may not be able to.

Adaptability:  In every intervention or program proposed for implementation, there will be essential components that must be unchanged regardless of local culture or requirements. However, measurement programs that can be readily introduced to new environments with easily accomplished customization of non-essential components are more successful.

Some aspects of IM4Q that should not be altered include its in-person interviews and the independence of the monitors and the system to achieve objectivity. The questions in EDE surveys that are tied to NCI and tap on quality of life (e.g., satisfaction, dignity, respect, choice and control, inclusion, and so on) are the essence of IM4Q. The IM4Q has the flexibility to change questions and samples based on the program goals and individual needs. Some stability and tracking data over time would be informative and could assist with decision-making. To make IM4Q more effective, some accommodations could be made. Video-conferencing interviews should be considered as an option to reach people in independent living and those living in rural areas of the state. Another important suggestion for the program is to increase the ability to access the real-time data for the relevant stakeholders to make real-time policy decisions.

Trialability:  Research establishes a link between the ability to pilot test a tool and successful implementation.  A trial period that can be reversed provides an organization with the opportunity to adapt and retool implementation based on the trial experience. 

Complexity:  There is an inverse relationship between the complexity of a proposed intervention and the likelihood of its successful implementation. The IM4Q EDE includes both open-ended and closed-ended questions rather than open-ended questions, which does add to the complexity of the interview process and requires monitors to have the ability to follow-up with participants to get an adequate response.  However, monitors are trained and observed to ensure that they can implement the interviews with fidelity. Additional complexity comes with in-person interviews conducted by a team.  This introduces greater coordination and scheduling difficulties, for example.  One of the challenges noted by participants was the increased difficulty in reaching and recruiting potential participants, although this is a challenge regardless of the data collection method.

Design quality and packaging:  When components of a measurement tool or program are well-designed and easily accessible to users, the intervention is more likely to be implemented successfully.  Flawed or poorly designed components can cause user dissatisfaction and/or lead to incomplete or inaccurate results. 

The IM4Q material consists of a manual developed by the state and Institute on Disabilities, Institute on Disabilities, Temple University and is available on the IM4Q website. Technical assistance is provided by advisors from Institute on Disabilities, Institute on Disabilities, Institute on Disabilities, Temple University to the local programs. Also, there are recruitment materials for participants and families as well as a website explaining the IM4Q process, purpose, and voluntary participation.  Annual training conferences and annual reports at the county and state level provide updates and other training about IM4Q on a regular basis.

Cost:  The implementation literature finds a negative association between the cost of the intervention and its success with implementation.  In addition to the cost of the tool and associated implementation costs, this part of the analytic framework also prompts consideration of “opportunity costs,” those items, projects, wages, or other financial obligations that cannot be pursued, completed, or prioritized due to IM4Q implementation. 

The IM4Q is funded through ODP.  As such, participants from ODP and AEs did not believe that the expense inhibited other important state initiatives.  Counties receive allocations each year from ODP and manage the actual costs related to fielding the IM4Q.  However, funding has been reduced making it difficult for local agencies financially.  In addition, the sample size has become smaller over time, in part due to funding issues.  The cost of implementing an in-person interview program that includes a team of monitors incurs costs such as travel expenses and wages that may not be present when doing phone-based interviews, for example. However, participants in case study interviews uniformly believed that having in-person interviews was an essential part of the IM4Q and provided information unavailable in telephone interviews.

Focus on Individual Needs and Resources:  In addition to the various components and characteristics of an intervention, the implementation literature identifies significant associations between specific environmental factors and successful implementation.  One construct in this domain is the degree of person-centeredness of an organization implementing a measurement tool.  The implementation literature indicates that implementation success will increase if implementers are knowledgeable about the individual needs of people receiving support as well as factors that hinder the provision of support.

This is a particular strength of the IM4Q.  Stakeholders’ input is at the core of the IM4Q. All monitor teams include a person with a disability or a family member of a person with a disability doing the interviews. It's opportunity for individuals themselves to provide information and improve their own quality of life; for family members to see what supports they need; residential and service providers and frontline staff have what they need to do their job effectively. Person-driven and evidence-based practices are  essential to make quality changes for everyone. The Considerations process addresses quality of life concerns stemming from individual needs through a systematic process that is documented in the states HSIS system. This process has feedback mechanisms to ensure that the appropriate people (such as the supports coordinator) are aware of and addressing the concern.  This increases the person-centered nature of the IM4Q.  Utilizing the data from EDE and Considerations to understand trends and themes can be one used as indicators to understand people’s needs.

Among the barriers individuals with disabilities encounter when participating in the program, communication is a concern for those administering the IM4Q. It requires the cognitive ability to understand and answer the questions independently. Some individuals may not be able to express what they truly want.  Interviewers do have interview aids such as icons and individual’s communication needs are identified before the interview.  In addition, there are some limitations to the program to reach non-verbal and non-English speaking individuals, although the EDE is available in Spanish. ASL interpreters are available to those who need them.  ODP is attempting to address these challenges through efforts to ensure that individuals in need of communication supports are receiving it (for example, ensuring those who need adaptive communication devices receive them).

Information Sharing and Shared Vision (Cosmopolitanism):  Another factor external to the measurement tool that affects implementation is the extent and nature of organizational support for staff to expand their roles to include keeping up with research, external training, and participating in professional groups. The implementation literature indicates that organizations that support these kinds of efforts are faster at implementing new practices.  

IM4Q is a state-specific measurement program.  Within the state, there is support in terms of technical assistance, quarterly meetings for AEs and their local IM4Q providers, and an annual statewide conference that includes staff from ODP, providers, AE staff, monitors, self-advocates, family members, support coordinators/case managers, and other interested parties.  The purpose of the annual statewide conference is to share information about IM4Q, including best practices and examples of how IM4Q has driven quality improvements.  In 2019, provider level reports were developed for any providers with 10 or more individuals receiving services participating in IM4Q.  This increases the ability of providers to make better use of IM4Q findings to improve their services and to learn from other providers.

External Policy and Incentives:  Motivation to implement an outcomes measurement tool is increased by policies or regulations that call for a focus on outcomes.  Although the IM4Q is initiated by the state, each county has its own plan to implement IM4Q.  Quality management includes providing IM4Q data at the county level which further drives policymaking at both the county and state level, a strength of and the IM4Q. Further, states’ DD agency leadership has been involved in designing and refining the HCBS tool.  The newly developed provider profile allows providers to assess their performance based on key indicators at the provider level, which can be used in combination with other sources to make informed decisions.

The 2014 revisions to the Final Settings Rule will require organizations and states to assure that they are achieving important outcomes in the areas of choice-making, self-determination, human rights, community access and inclusion, and person-centered strategies of support, among other requirements. The new rule does not prescribe the method of outcomes measurement.  The IM4Q’s EDE (and the use of the NCI) provides system level information focused on individual quality concerns through the Considerations process meeting the CMS mandate that the required outcomes areas are addressed in a person-centered process.

Framework Domain: Inner Setting

Inner setting addresses organizational attributes including structure, networks and communication, culture, implementation climate, tension for change, compatibility with the intervention, incentives and rewards, goals and feedback, learning environment, readiness, available resources, and access to knowledge and information.  This study was not designed to examine the capability of one or more specific organizations to implement the OM tool effectively. Instead, it looks at the characteristics of the tool that facilitate or hinder implementation across implementation sites. There are, however, some general observations from our experience that apply to this domain.

IM4Q is well understood and appreciated at the state level and is a priority for ODP. The implementations of IM4Q depends on the teamwork and commitment of various levels of local, regional, and state organizations. Procedures have been developed to be able to implement the program effectively.

The constructs identified in this domain include individual knowledge and beliefs about the intervention, individual stage of change (degree of progress toward skilled implementation), individual identification with the organization, as well as other personal attributes such as motivation or flexibility.   Similar to the previous domain, this study was not focused on how the characteristics of specific opinion makers, leaders or stakeholder’s mediate implementation. 

The state staff interviewed for this case study report that their leadership places high importance on quality assurance and the use of data to make policy decisions, and, therefore, has invested in the IM4Q program.  The IM4Q program is woven into Pennsylvania’s quality monitoring and assurance programs.  For example, data from the EDE and Considerations are entered into the state’s HCSIS.  This ensures that individual needs are addressed by case managers and allows a larger systemic view of trends and need areas.  This process led to the identification of a lack of access to communication.  Access to communication became an area of focus based on the state’s quality assurance program.  Other areas of focus have been making sure people who desire to register to vote have an that they then get to vote; the unauthorized opening of a person’s mail does not occur; an promoting the employment of people with disabilities.

Framework Domain: Process  

The CFIR developers identify the “process” domain as the most challenging domain in implementation science to evaluate due to several theories on implementation that range across topics including total quality improvement and management, integrated support, complexity theory, and organizational learning (Dramschroder et al., 2009).  The CFIR framework focuses on four constructs that are commonly found, either explicitly or implicitly, in a range of theories and frameworks:  1) Planning, 2) Engaging, 3) Executing, and 4) Evaluating.

Planning:  The level of advanced planning for intervention is positively associated with the success of the intervention. 

The IM4Q has been in place for nearly two decades.  Planning consists of preparations for the annual monitoring surveys.  Because the systems and processes are in place and challenges are addressed as needed, this is a strength area for the IM4Q.  One example of this is the recognition that communication about the IM4Q to potential participants and their families needs to be tailored to address both the change in technology (such as cell phones versus landlines), the increasingly diverse population, and helping people who live in individualized settings understand why participation is of value to them.

Executing:  To determine the components essential to the fidelity of the IM4Q, the essential components were identified as:

• The independent and local nature of the IM4Q interviewing process. Every person interviewed identified this as a key component of the program that should be maintained.
• In-person interviews that were conducted by teams of people that included people with disabilities and/or family members.
• The Considerations process that allowed for opportunities to address quality of life issues at the individual level.

Reflecting and Evaluating: 

The IM4Q process has a steering committee comprised of stakeholder groups that meets quarterly to review the IM4Q process, findings, and to address challenges that are identified through regular meetings with AEs, local coordinators, and technical assistance.  In addition, there is an IM4Q Management Committee that addresses and advises on operational aspects of IM4Q.