Outcome Measurement Program Case Studies

Case Selection

Cases (measurement programs in HCBS) were selected by project staff in consultation with the Project Advisory Committee (PAC) based upon key characteristics (Thomas, 2011). Components considered in selection include cases that illustrate a variety in:

1. measure focus  (e.g., organization, region or state);
2. methods used to obtain data (e.g., administrative dataset, in-person interview by contracted personnel, in person interview by state or organization personnel, or use of individuals with disabilities in the interview process,);
3. type and comprehensiveness of training involved to prepare data collectors;
4. sample size within the program;
5. target population;
6. integration of proxy or non-proxy procedures;
7. scope and potential for scale-up of the outcome measurement program;
8. fee for service versus managed care LTSS  environment; and,
9. depth and alignment of the outcome measurement domain areas with NQF and the results of Phases  #1 and #2 of the RTC/OM.

NCI-IPS was chosen because it is a national program used by nearly all of the states’ Developmental Disability (DD) agencies as a state system-level measurement tool.  It meets the above criteria in terms of its comprehensive training and technical assistance program, defined proxy-response procedures, and alignment with NQF domain areas and RTC/OM findings.  NCI is used in states that have managed care LTSS. However, no managed care entities or states with managed care participated in this case study. 

Instruments and Sources of Data

Multiple sources were used to collect data on NCI.  Sources of information included key informant interviews of stakeholders with knowledge of the design, administration, and implementation of the National Core Indicators including, 3 state agency administrators charged with managing their states’ NCI programs, 2 trainers, and 5 surveyors from three different entities (two contracted survey and research firms and one university-based program).  We have also interviewed stakeholders who use the program to make policy and program decisions based on outcome measurement (OM) results, including state and regional administrators and providers who implement HCBS programs. 

Data probes and questions were designed with an open-ended format to offer the opportunity to identify new implementation constructs and insights that have not been described in the existing literature related to HCBS outcome measurement implementation.

Other sources of data included:

1. an examination of peer-reviewed literature concerned with the fidelity of implementation and implementation science as well as current research on quality of life measures and constructs, inclusive of recent publications and endorsements of OM tools by the NQF and CMS rules regarding HCBS settings and practices;
2. OM program documents, including marketing materials, protocols and measurement reports, as well as other written materials about the OM program such as evaluations, surveys and periodic updates of the program tool and procedures;
3. descriptions of policies, regulations or other system changes resulting from OM program activity; and,
4. observations of surveyor training and simulated implementation.

An interview guide and protocols were developed to guide the data collection (See Appendix C).  Protocols varied based on the category of persons interviewed.  One standard protocol (Protocol A) was used to interview OM program staff, state administrators, and staff from organizations using the OM tool. Slight variations were made in Protocol A to fit the context of each group.  A second protocol (Protocol B) was developed for other content experts in outcome measurement familiar with the selected OM and the state or region in which the OM program was studied. RTC/OM research staff, national advisors, and staff from NIDILRR and ACL reviewed the protocols before implementation. 

We defined implementation fidelity as “the extent to which the critical components of an intended program are present when that program is enacted” (Century et al. 2010).  Adherence to this definition required that the study team identify the critical or essential components of each of the HCBS OM programs.  We developed a query form that OM program senior staff and trainers were asked to complete to identify those components perceived as essential or critical to the success of the program.  These forms were then reviewed with program administrators to clarify decisions about the essential or non-essential nature of the components identified. 

Planning for Site Visits    

The project staff held a series of planning meetings with the OM program staff.  These meetings were used to identify and request relevant OM program materials and to identify sites with plans for scheduled implementation activities.  Once potential visit sites were identified, the OM program staff made initial requests to obtain permission for relevant organization staff to participate in the case study.  After the initial permission was granted and site contacts were identified, the team worked with the site visit contact to set up the visit interview and observation schedule.  A site visit was made to California to observe training and conduct interviews.  The site visit activities were conducted over 3 days.  Also, phone interviews were conducted with one state agency administrator and two interviewers/interview mentors in Minnesota.  Phone interviews also occurred with interviewers in Florida (2) and Oklahoma (1).