RTCOM Briefs

Introduction

Adults with disabilities who meet national and state eligibility requirements and require long-term services and supports (LTSS) receive these services through a variety of programs overseen and funded by the Center for Medicaid and Medicare services (CMS). In order for states to receive CMS funding, they are required to measure the impact of the services provided to people with disabilities as well as the quality of those supports. In alignment with the CMS oversight process of 1915(c) waivers, the Home and Community Based Services (HCBS) Settings’ Final Rule (CMS, 2014) and the National Quality Forum’s (NQF) HCBS measurement framework (NQF, 2016), greater emphasis has recently been placed on evaluating the outcomes experienced by people with disabilities as well as their quality of the services they receive.

Measuring service quality and the outcomes of HCBS recipients is far from a coherent, consistent, process. It has been approached in different ways across states and service providers. Currently, a wide variety of measurement tools are being used to evaluate these outcomes. Tools in use vary significantly with respect to the approaches they take to data collection (e.g., who collects data and who is interviewed), the psychometric characteristics of the tools (including their reliability and validity), and the degree to which they are person-centered, i.e. focusing on the HCBS recipient’s desired life outcomes. Some tools focus on quality assurance, while others are centered on the personal outcomes experienced by people with disabilities. The degree to which existing measures tap into the HCBS outcome measurement domains and subdomains established by the NQF (2016) and validated by the Research and Training Center on HCBS Outcome Measurement (RTC/OM, 2016) also varies considerably.

In addition to the variety of tools and measures used to determination service quality and the outcomes of HCBS recipients, little consistency exists in how states determine whether people are entitled to HCBS as well as the service packages/case mix for which beneficiaries are eligible. A small number of states use well developed and validated eligibility assessment tools (e.g., The Supports Intensities Scale – AAIDD, 2015). Some states use assessments that, while possessing empirical evidence to initially support their use are outdated (e.g., Inventory of Client and Agency Planning). Still other states employ measures used for eligibility and funding determination on which there is little to no evidence of their reliability and validity.

As a result of this state of affairs, a wide variety of stakeholders are in need of information relevant to measurement in HCBS. This includes states, providers, non-governmental organizations (e.g., CILs and The Arc), researchers undertaking work in HCBS related areas, and the recipients of HCBS themselves. An understanding of current measurement systems and their characteristics has the potential to support stakeholders to better understand current approaches to HCBS measurement, their strengths and limitations, and to make more informed decisions with respect to the measures they use and the manner in which their results can be interpreted.

The primary mission of the University of Minnesota’s Research and Training Center on Home and Community-Based Services Outcome Measurement (RTC/OM) is to conduct and research and technical assistance to improve HCBS outcome measurement. One of the Center’s major activities over the past several years has been to perform an environmental scan of existing HCBS outcome instruments, map the instruments and their items to a revised version of the National Quality Forum’s (NQF) HCBS outcome measurement framework (NQF, 2016), and assess the extent to which existing tools are both person-centered and demonstrate psychometric data to support their use. The HCBS Outcome Instrument and State HCBS Tools relational databases were developed as a result of this work. These databases that are part of the RTC/OM’s website are designed to assist users to better understand the measures currently used in the field to be able to make more informed decisions with respect to their potential usefulness in addressing their needs.

Development of the HCBS Outcome Instrument Database

The HCBS Instrument database is an open access repository of information about instruments/ tools developed to assess HCBS quality and outcomes of people with different types of disabilities. It currently contains information on over 130 existing instruments identified as assessing HCBS quality or outcomes. Center staff mapped each item from each instrument onto the domains and subdomains of outcome quality identified by the revised NQF Framework (see Figure 1). The database was designed to assist users (e.g. measure developers, program administrators, policy makers and researchers) to identify tools that fit their intended purpose and/or verify a need to develop new tools and measures. It is designed to benefit researchers and providers to strengthen the rigor of evaluation and research associated with HCBS policies, programs, and interventions.

Revised NQF Framework for Home and Community Based Services Outcome Measurement. 13 domains. 2-7 subdomains per each domain. The domains are:

• Choice and Control
• Human and Legal Rights
• Transportation
• Person-Centered Planning and Coordination
• System Performance and Accountability
• Caregiver Support
• Community Inclusion
• Workforce
• Employment
• Holistic Health and Functioning
• Consumer Leadership in System Development
• Service Delivery and Effectiveness
• Equity

Instrument Search and Item Coding

A comprehensive search process was conducted to collect all available instruments being used to assess HCBS outcomes beginning with the most widely used tools (e.g., National Core Indicator surveys). Next, an extensive literature search was conducted and a review of instrument compendia related to recovery, mental health, disability, caregiving, and traumatic brain injury undertaken to locate relevant instruments. Instrument attributes including purpose, population, administration mode and method, person-centeredness of items, psychometric data available, and other instrument-level characteristics were coded for each tool. Items from each instrument were double-coded by two Center staff persons based on the operational definitions of NQF domains and subdomains, respondent types, response options, and person-centeredness. An item was considered person-centered if it allowed the HCBS recipient to directly express whether a preference, desire, or need was met, or articulate a level of satisfaction or emotional state. Notably, not all items were able to be mapped onto the NQF framework. See actual item coverage per domain.

Psychometric Background of Instruments

Understanding the development and testing history of an instrument is key to selecting appropriate, high quality measures. One of the contributions of the instrument database is that, when available, it provides users with information with respect to the psychometric evidence (i.e., reliability and validity) supporting the use of an instrument or measure. Technical reports, academic journal articles, and book chapters within which the psychometric properties of instruments in the database were located through multiple databases (e.g., Cochrane Library Google Scholar, MEDLINE, PsycINFO, PubMed). The following psychometric information (as available) was coded and linked to corresponding instruments:

1. Sample characteristics used in reliability and validity studies (e.g., sample size, age, gender, region/country, and disability type
2. Sampling method: probability or non-probability sampling
3. Administration: interview or survey via in-person, phone, mail, or web-based.
4. Reliability: internal consistency, test-retest, split-half, and other
5. Validity: construct, content, criterion, face, measurement invariance, and other

The HCBS Instrument database is the only searchable database of which we are aware that systematically applies the National Quality Forum’s HCBS Outcome Measurement Framework to an extensive collection of instruments. This makes it possible for users to compare instruments with respect to their coverage of an HCBS outcome measurement framework that has had its content validity assessed with a variety of stakeholder groups increasing the likelihood that users will select an instrument that fits their measurement purposes and has evidence of its psychometric quality. Moreover, mapping the items of which instruments and measures are composed onto a framework makes it possible to identify those areas in which there are gaps in outcome measurement at either the organizational/provider or system level.  This information can be used to increase measurement effectiveness and capacity. In addition, the database makes it possible to understand the extent to which existing tools employ a person-centered approach to measurement.

Development of the State HCBS Assessment Tools Database

After completing initial development of the HCBS Instrument database, RTC/OM staff undertook a systematic process designed to explore, collect, and confirm information related to the outcome measures and instruments used in state HCBS waiver programs. As a result of this effort, the State HCBS Assessment Tools database was created. The State Tools Database was developed to help users obtain a better understanding of how various HCBS programs across the nation are using outcome measures. The database outlines the use of two primary types of assessments: those focused on quality of services and the outcomes experienced by program recipients, and assessments centered on eligibility, funding, and support needs.   

To develop the database, a two-phase data collection effort was conducted for each state. During the first phase, RTC/OM staff reviewed approved state waiver applications and contacts, HCBS final rule transition plans, and state documents and web pages to identify and code information about both HCBS programs currently available within the state and the measures/assessments that were used to determine eligibility, funding, and case mix.  Following completion of the initial data gathering efforts, key officials within states were contacted to verify this information and determine any changes or additional assessments states were employing. This database is periodically updated with new information as it becomes available.

The State Tools database provides a unique contribution to the HCBS measurement field. It allows users to compare the types of outcome and eligibility assessments used in various waiver programs within a state. In addition, comparisons can be made across states implementing similar waivers. Although individual states provide information about the assessments they use, this is the first time it has been possible for users to find information across all states within a single database. With the multitude of approaches to measurement used by states, the database is a valuable resource to compare and further the field of HCBS measurement.

Using the Databases

Using the HCBS Outcome Instrument Database

The following section provides a basic overview of the structure of the HCBS Instrument Database. It also explains how, when, and why a user might desire to access this information. Figure 2 displays the major areas of the web-based relational database.

A flowchart that illustrates how you can move through the web-based databases on the RTC/OM website. There are two main tools:

1. HCBS Outcome Instruments which has search and filter functionality
2. State HCBS Assessment Tools which is an interactive state map

From the HCBS Outcome Instruments, you can review specific information pages such as domains and psychometics.

From the State HCBS Assessment Tools Interactive map you can go to:

• The search and filter page
• The Waiver Program Pages
• A list of the individual State HCBS Programs
• Specific information pages such as domains and psychometics

Searching the Instrument Database

The HCBS Outcome Instrument database includes a search function that can be used to locate specific HCBS outcome instruments through searching by the instrument name, acronym, or keywords. Users are also able to search via the use of specific filters (e.g., NQF domain or subdomain) to narrow results to a specific outcome concept of interest. 

There are several other filtering options to further narrow the search parameters to match the user’s purpose including:

• Respondent types: including persons with disabilities, caregiver, family member, provider, and proxy;
• Population: including age-related disabilities, intellectual and developmental disabilities, mental health conditions, physical disabilities, and traumatic brain injury;
• Administration Method: interview or survey;
• Administration Mode: in-person, mail, phone, and web-based;
• Type of Reliability Evidence Available: including inter-rater reliability, internal consistency, split-half, test-rest, and other reliability evidence;
• Type of Validity Evidence Available: including construct, content, criterion, face validity, and other validity evidence; and
• Psychometric Data Source, including book chapter, peer reviewed article, and technical report.

Individual instrument characteristics

After selecting an instrument in the search results, an individual page for each instrument, including information about its intended purpose, psychometric characteristics, and background will be uploaded. Information for most instruments also includes: respondent types, administration, item counts, and intended population as well as citations to which users can refer for additional information on an instrument. Another important piece of information included on each instrument page are the National Quality Forum domains and subdomains covered by an instrument. For each domain and subdomain the number of items an instrument possesses that assess outcomes within that area are provided. Some instruments (e.g., the NCI-IPS survey) cover multiple domains, while others (e.g., Community Integration Questionnaire-Revised) focus on a single domain. Users should contact the author and/or publisher to obtain the permission for the use of the full instrument.

Psychometric Evidence

It is important to note that not all HCBS outcome instruments/ measures currently available possess the evidence to support their reliability and validity. For others, although such data is available, it indicates that the measure(s) in questions fail to meet minimal psychometric criteria for usage established by national organizations (e.g., NQF Measure Evaluation Criteria, 2017; Centers for Medicare and Medicaid Services, 2018, p. 197).  If no psychometric evidence is shown for a particular instrument, either the instrument/ measure has not undergone psychometric testing, or evidence to attest to its reliability and validity has not been published (or made available to center staff). Psychometric evidence needs to be re-established whenever significant changes are made to an instrument or measure, it is going used for population on which it was not initially validated, or in a setting different from that for which it was designed. We recommend that sufficient reliability and validity evidence be a key criterion when selecting HCBS outcome measures.

Using the State Tools Database - Discovering How States are Using HCBS Assessments

The main entry page into the State HCBS Tools Database includes a map of the United States. A description of the methods used to create the database and a link to the search and filter page are available on this page. This page is particularly useful during the early stages of outcome measurement exploration or when users are interested in learning about the measures a specific state uses to evaluate one of its waivers or make eligibility, care-mix, and/or funding decisions.

Searching the State Database

Once a user enters the State Database, information regarding the tools that individual states use to assess HCBS outcomes and determine eligibility and case-mix can be located in one of two ways. If a user is interested in a specific state, clicking on that state on the map will take them to that State’s page (See figure 3).  The page will start with a list of all identified HCBS waiver programs within the state of interest, characteristics of the population(s) served, and a list of assessments/measures each program. Alternatively, by searching the HCBS assessment tools the user will be brought to the primary search and filter component of this database. Here, users can elect to search on the basis of: 

• State: A list of all states is available for users to select individual or multiple states to narrow the waiver list to specific states;
• Instrument Used: An alphabetical list of all known assessments by type (quality/outcome or eligibility/support) is included
• Waiver Type:  A list of waiver types (e.g. 1915c) to choose specific types of HCBS waiver programs; or
• Population and Age Group: This filter can be used to narrow down the list of waiver programs to only those serving specific populations (e.g. physical disability) or age groups (22-64 years).

The State HCBS Assessment Tools area of the RTC/OM website includes the results of the center’s effort to identify the assessments used in HCBS programs serving people with disabilities across the United States. The map represents the verification efforts. The map legend contains four verification types

• No information is currently available
  • Rhode Island is the only state where there is no information currently available.
• Information based on state waiver applications only. No verification provided by state.
  • South Dakota, Kansas, and Maryland have information based on state waiver applications only.
• Partially Verified by State Officials
  • Utah, North Dakota, and Vermont have information that has been partially verified by state officials
• Verified by State Officials
  • Forty-four states have been verified by state officials.

In addition to the filters, there is also a search field for more specific criteria. Selection of any combination of the filtering components will reduce the list of HCBS programs to only those matching the filter criteria. Users may use various combinations to fit the specific review criteria and use the specific waiver and assessment pages to either review how specific programs are using assessments or learn more about a specific assessment.

Learning More About Specific Waiver Programs and Outcome Assessment

Once a state and HCBS waiver/program of interest has been identified using the above options, selecting the waiver of interest will bring the user to a page with details of the assessments used within that program. This page includes a list of the two types of assessments (Eligibility/Support Assessments and Quality/Outcome Assessments) and how those assessments are used (i.e. disability populations, age group; sampling strategy, administration frequency, and typical sample size). Clicking on an individual assessment will bring the user to an additional page listing additional assessment characteristics.

Conclusion

The HCBS Instrument and State Tools databases each make unique and valuable contributions to the HCBS measurement field. Both can be used as part of a process to learn about HCBS-related assessments being used and as part of a strategy to select a tool to fit the user’s purpose.

Recommendations

Recommendation 1

State HCBS program administrators and policy makers should incorporate a review of these databases when considering changes to the use of HCBS assessments tools in their programs.

Recommendation 2

Instrument users should consider the psychometric properties of instruments and measures including their reliability and validity, when selecting from among existing measures that fit their measurement concept of interest

Note: Databases are routinely updated as new measures or new information becomes available.  Please contact RTC/OM staff at rtcom@umn.edu with suggested additions, new information, or corrections.