Outcome Measurement Program Case Studies

Components to Ensure High Fidelity in the Implementation of HCBS Outcome Measures

The NCI-IPS has several components that support implementation fidelity (See Appendix B for detailed information about the Alignment of the NCI-IPS components with CFIR).

These include a standardized in-person training protocol that provides opportunities to practice interviews, videos, and manuals.  In all of the organizations interviewed, new interviewers are encouraged to both observe interviews and to be observed before interviewing on their own.  These organizations also had supervisors or experienced interview mentors who observed interviewers regularly.  Interviewers reported having access to technical assistance in the field and access to videos to refresh their skills when the NCI was in the field. 

In general, those interviewed for this case study, regardless of position, agreed that following the interview protocol was important to collecting good data.  However, entities had different processes when the data were collected.  Some provided a “hands-on” approach in which interview coordinators/supervisors reviewed the data for quality before it was entered in ODESA (the central NCI database).  They reported being able to clarify unexpected answers or missing data with an interviewer while the interview was fresh in the interviewer’s mind.  For other entities, the data was uploaded and those interviewed were unsure of what happened to the data once it was entered into the system. 

Usefulness of Data

The most significant critique of the NCI program came from state DD agency staff.  They would like to be able to get their data sooner to help with their agency’s decision-making about policy.  For some states, the NCI is an integral part of their quality improvement strategies and is woven into their system (e.g., Pennsylvania).  Other states participate in NCI every year, but state staff doesn’t believe that it is used to its fullest extent possible to make policy decisions.

Strengths of the NCI Identified in the Case Study

• NCI-IPS provides a train-the-trainer approach for organizations that are contracted with states to conduct interviews.
• NCI-IPS provides technical assistance on survey implementation, data collection, and support to assure state practices align with NCI protocol.
• NCI-IPS provides state reports that enable states to evaluate system performance.
• NCI-IPS enables states to make data-driven policy and programmatic decisions based on the reported data.
• NCI-IPS measures have been selected for inclusion in Medicaid-CHIP Adult Core Measure Set (2020).

Challenges of the NCI

• Data collected through the NCI-IPS is not designed to be used for improving supports at a provider or an individual level.
• NCI-IPS is not designed for longitudinal data collection, so changes over time using the same sample are not measurable.
• Participants, particularly in less populous states, can experience survey fatigue from repeated annual data collections. Some smaller states administer the NCI-IPS in the alternating years to address this challenge.

Factors, Characteristics, or Components that May Strengthen or Deter Effective HCBS Outcome Measurement Identified in NCI

Positive drivers include:

• State agency leadership’s commitment to the measurement process to drive quality improvement for LTSS and HCBS.
• Presence of an external mandate that requires outcome measurement, such as state or federal regulations.
• States’ ability to add items to measure state-level goals.
• The ability of states to measure their performance against other states regionally or nationally and to learn from each other.
• Measurement programs that are designed to assure fidelity of implementation.
• NCI-IPS includes items identified as indicators that focus on key HCBS outcomes domains.
• Sufficient levels of training and technical assistance are significant for implementing and interpreting NCI data.
• The development of state-level reports and access to state-level data for further analysis.

Deterrents to implementation include:

• State-level implementation means that buy-in from providers and individuals is difficult (See Appendix B).
• The time between administering the survey and receiving the results is too lengthy.
• Some concepts and questions on the survey instrument are too complex for some participants leading to the possibility of unreliable or missing data.
• The aims and purposes of the tool are not readily apparent to some providers and HCBS recipients.
• Changes to the instrument itself, particularly the administrative section, can be challenging at the state level because state data collection doesn’t always match NCI data categories. State respondents report that this causes states to have to re-work their programs to pull data from their databases.

Recommendations:

1. To increase the utility of the NCI data and decrease the amount of time from data collection to availability to the states.
2. Provide additional or different training for experienced interviewers to increase their interviewing skills, particularly for those interviewer staff who are serving as lead or mentor interviewers.
3. Ensure that interviewers understand the purpose of the NCI and how their state uses the data. Interviewers who understand how the data is used will be able to answer participant questions, which may encourage participation.
4. Continue to test and report the psychometric properties of NCI items.