Outcome Measurement Program Case Studies

Methodological components need to ensure measure administration fidelity

A key element of fidelity is the full implementation of critical components of an intervention (Century et al., 2010). The level of administration fidelity of a particular measurement tool can be assessed by examining the key components of the tool’s design and observing the extent to which these components are implemented. POMs® content experts we interviewed as well as our own examination of the POMs® tool and process indicates that the key components of the POMs® are thoroughly implemented as the designers envisioned.

Implementation fidelity, however, does not assess the extent to which a measurement tool is measuring relevant or appropriate constructs. For example, a measurement tool may be fully implemented as the designers intended, but may not measure the most relevant or targeted HCBS outcomes. Therefore, the particular tool may be an inadequate or inappropriate instrument to measure HCBS quality outcomes. The NQF (2016) report on measurement gaps in HCBS provides a comprehensive framework for the domains of HCBS quality that are important to measure. Using the NQF domains as a framework to select an instrument or combination of instruments to measure HCBS quality will lead to relevant data that can be used to assess program quality, in addition to fidelity of implementation. The NQF report does identify the POMs® as a source measure for several of the domains identified as important.

Strengths and Challenges in Measure Administration

Strengths of instruments like the POMs® in measuring HCBS outcomes:

• Focusing attention on important life outcomes in measurement is essential to assure that stakeholders pay attention to these outcomes when funding, planning, providing, and evaluating supports and services. Focusing on personal outcomes gives some assurance that the CMS Quality Strategy Goal #2 that calls for person-centered strategies to fully engage people and their families in the design, delivery, and evaluation of care (CMS 2016) will be implemented.  Measures that are part of this assessment tool  address the NQF recommendation to use some measures that provide unique, contextualized information relevant to the individual’s situation as well as the NQF recommendation to build on existing measures.
• Offers a well-designed and accessible process to measure the quality of life domains and aspects of person-centered practices despite the complexities the process entails. This is of great importance and relevance to the changes CMS is seeking to promote with its revision of HCBS requirements.  The use of well-designed decision guidelines and competent trainers/coaches is critical to anchoring reliable and accurate assessments of quality outcomes.
• Employs a process that engages employees, especially frontline staff, in cohorts for learning, dialogue, and critical thinking that supports continual positive change in their day to day practice. The training and decision-making process provides many opportunities for the exchange of ideas, practices, and exploration of support quality. This type of interaction and reflection is a powerful resource for building individual and organizational learning about quality.  It also provides opportunities for peer connection and team building that are often lacking in the decentralized service settings common in HCBS. This process is very different than a process where external evaluators talk to people or where a survey is conducted with a rating scale.
• Provides a conceptual model of person-centered quality outcomes that can be shared across organizations on a local, regional or statewide basis to promote a common vocabulary on personal outcomes. Such measures can help diverse and decentralized organizations to work cooperatively to calibrate the assessment of quality outcomes for their context. 
• Stresses the importance of each person expressing his or her version of quality life. Its emphasis on individual variability and a personally constructed vision of a quality life enables the tool to work across diverse settings and populations.
• The POMs® PORTAL provides a tool to analyze individual results and produce reports that are specific to one person as well as providing mechanisms for aggregating results and generating reports that can be useful in identifying systemic patterns.
• Provides a holistic and integrated way to understand the life experience of persons served.
• Offers a concise and psychometrically sound collection of indicators of individual quality of life outcomes.

Challenges of instruments like the POMs® in measuring HCBS outcomes:

• Requires initial and periodic training and recertification of interviewers to collect and enter data in standardized manner to aid in replicability and evidence-based decision making.
• It is most useful when direct support staff is engaged in the process. This can be difficult, given the high turnover and vacancy rates among direct support staff.
• A limited set of outcome areas may not cover all domains identified by the NQF (2016).
• Requires significant time commitment to complete each protocol.
• While there is some capacity to use the tool for inferential statistical analysis, constructing samples, and arranging for interviewing representative samples presents logistical challenges.

Factors, characteristics, or components of OM programs that strengthen or deter effective outcome measurement of community living and participation programs:

Positive drivers include:

• Tools that focus on people in a respectful and integrated manner, are meaningful and understandable by program staff and others who need to learn about person-centered domains of quality important to people with disabilities.
• Agency leadership involvement and visible commitment to the measurement process stimulates staff to engage with quality measurement and improvement.
• Involving frontline staff in learning tool outcome areas and discussing results fosters a commitment to follow-up on the results through specific changes in day to day supports.
• Organizational culture and support that is outcome-focused and person-centered increases the potential for full engagement with quality measurement at all agency levels.
• The presence of an external mandate that requires outcome measurement, such as a state regulation, coordinates implementation tools prioritazation and selection, thus improving comparability across locations.
• Measurement programs that are packaged attractively, use accessible language and strive to be concise and intuitive generate engagement of staff.
• Indicators that focus attention on meaningful life outcomes provide a window into how supports provided by staff contributes to outcomes of peoople with disabilities positively or negatively.
• Feasible cost structure and external financial support for quality improvement activities are especially important to smaller agencies that may have fewer resources than larger organizations.
• Support for regional communities of practice organized for learning and mutual support in HCBS quality improvement support successful implementation.
• The ability to scale up the measurement program at a reasonable pace promotes buy-in.
• Adequate levels of training and technical assistance provided by well-informed trainers and coaches are very important for implementing measure and interpreting results for effedtive decision making and quality improvement.

Deterrents to implementation include:

• Top-down processes that exclude frontline staff diminish staff engagement with outcome measurement and quality improvement.
• Insufficient resources to engage the entire organization will limit the potential for quality improvement services.
• Lack of commitment and attention from organizational leaders to quality measurement communicates that quality is not an organizational priority.
• Measures that conflict with other measurement tools/systems or replicate existing outcomes measurement efforts diminish implementation success.
• Lack of mandates to implement quality measurement deter implementation and result in difficulty in measuring quality across local, state, and national systems.
• An organizational culture that views measurement as a burden vs. a tool for change deters effective implementation.
• Organizations where leadership and staff are insufficiently informed about person-centered support, or that do not see person-centered support and individual outcomes as a priority are less likely to implement tools focused on individual outcomes effectively. Changes to individualized support and to the strategic direction of an organization based on the results of such outcome measurement (OM) programs and tools are also less likely.

Person-centered measurement tools, such as the POMS®, that measure the domains of an individual’s quality of life are essential to measuring and moving support in the direction that CMS has articulated in the HCBS 2014 Final Settings Rule change. The POMs® and similar tools that directly engage people in talking about their life experiences will also support the CMS quality strategy goal #2, calling for greater involvement of people and their families in support planning and delivery.  Priority measures, especially those capturing the experience of the people who use HCBS, can serve as a starting point to measuring critical aspects of HCBS quality.