Outcome Measurement Program Case Studies

Appendix C
Case Study Protocol

Case Study Purpose

The RTC/OM was created with federal funds to assist in the development of effective HCBS outcome measurement tools, and this case study will make an important contribution to that goal. Several case studies will be undertaken to examine how commonly used HCBS outcome measurement tools are implemented, and to explore factors that influence their implementation. Through cross-case study analysis, the research team will seek to identify common measures and best practices in implementation to inform stakeholders on the effective measurement of HCBS.

Process

With input from the RTC/OM National Advisory Group (NAG), the research team identified several nationally recognized HCBS outcome measurement tools to study that are frequently used by states and organizations that operate HCBS.

In partnership with the organizations that offer the measurement tools, the research team will plan and schedule study activities including: identification of critical program components, collection of materials that describe the tool and its implementation procedures, visits to observe aspects of tool implementation, and interviews with key informants including the tool developers and site leaders, tool users including service organizations and government agencies, advocates for people receiving HCBS and people receiving HCBS.

Methods

Research Questions (Please note that research questions may change in response to certain project findings):

What are the strengths and challenges of various outcome measurement programs?
How do these strengths and challenges impact measure administration fidelity?
What methodological components need to be in place to ensure measure administration fidelity in the implementation of HCBS outcome measures?
To what extent have known factors that are important to fidelity been attended to in the programs reviewed (e.g., interviewer training, criterion testing, protection against/identification of response bias)?
What are the similarities and differences in implementing various outcome measurement programs?
What are the likely/potential implications for validity and reliability of data gathered that might derive from the differences?
What factors most facilitate or distract from effective implementation of programs regarding community living and participation outcome measurement?
What methodological components need to be in place to ensure measure administration fidelity in the implementation of HCBS outcome measures?
What are the strengths and challenges of various outcome measurement programs and how do these impact measure administration fidelity?
What are the similarities and differences in implementing various outcome measurement programs?
What factors most facilitate or distract from effective implementation of programs regarding community living and participation outcome measurement?

Data Collection

Identification of critical program components in partnership with tool developers:

Review of written materials including: procedure manuals, site leader selection process and training materials; marketing and end-user materials; description of technology used to support implementation and measurement; descriptions of technical assistance provided to measurement sites; description of methods for documentation of measurement data, description of methods for analysis and reports or feedback of measurement data, and any current technical and research reports focused on the measurement tool.
In-depth interviews with key informants;
On-site observation;
Review of draft case study findings with the measurement organization; and,
Development of a written case study report.

Key Terms

Critical components: These are a measurement program’s elements that are essential to achieving its goal of effectively measuring HCBS outcomes.
Critical process-facilitative components: A sub-category of the critical components of a measurement program that describe the tool implementers’ (facilitator, site leader etc.) behavior and interactions that are essential to effective measurement of outcomes such as: engaging participants in discussions; providing clear directions; providing accommodations as needed to participants; using terms and phrases that people understand, supporting people to speak freely.
Critical process-participant engagement components: A sub-category of the critical component of a measurement program that describes the participant engagement behaviors essential to effective measurement of outcomes such as speaking freely (no fear of reprisal); active participation in interviews and discussion groups, etc.
Critical structural- knowledge components: A sub-category of the critical components of a measurement program that concern the knowledge and experience requirements of the site leader/surveyor.
Critical structural-procedural components: A sub-category of the critical components of a measurement program that provides a format and organizational structure to the program such as implementation manuals, sampling plan, training curricula, site leader/facilitator selection criteria site leaders and team members, tools and materials used by participants, measurement criteria etc.
Fidelity of implementation (FOI): The extent to which the critical components of an intended program are present when that program is enacted. It is concerned with whether a program was enacted as intended. (Framework and key term definitions adapted from Century, J., Rudnick, M., & Freeman, C. (2010). A framework for measuring fidelity of implementation: A foundation for shared language and accumulation of knowledge. American Journal of Evaluation, 31(2), 199-218.
HCBS outcome measurement: A specific process or activity that yields a measurement of an HCBS outcome.
HCBS outcome measurement program or tool: A coordinated process through which a state intentionally measures the quality of its HCBS. This process may use a variety of methods that may or may not include the use of a standard instrument or interview protocol.
HCBS personal outcome: A measurable difference in an HCBS recipient’s life as a result of an HCBS facilitated intervention.
HCBS population outcome: A measurable difference in the lives of a group of HCBS recipients as a result of an HCBS intervention.
Implementation research: Research that examines the implementation of specific programs, models, or interventions.
Measurement Program Partner: An organization or consortium that maintains and implements a specific HCBS Measurement Program.
Non-essential components: These are program components that are not essential to fulfilling the goal of measuring HCBS outcomes.
RRTC/OM Advisory Group: Persons of national influence who are exceptionally knowledgeable about HCBS and the role of RTC/OM including policymakers at the state and federal level, advocates, quality measurement program staff experts and methods/measurement experts.