Outcome Measurement Program Case Studies

The IM4Q program grew from a 1997 recommendation from Pennsylvania’s Office of Developmental Programs (ODP) Planning Advisory Committee to develop a mechanism to monitor the quality of life of individuals receiving services under the auspices of the ODP.  A sub-committee of people who use ODP services, family members, advocacy groups, providers, Administrative Entities (AEs; counties and other local government units charged with administering DD services on behalf of the state), direct support professionals, union representatives, the PA Developmental Disabilities Council, the PA Disability Rights Network, and ODP staff recommended that quality monitoring should be independent leading to a pilot program being conducted in 1999-2000.  This pilot program, summarized in the 2002 State Report of Independent Monitoring for Quality along with recommendations from the IM4Q Steering Committee was the basis of developing an action plan (2003 Quality Action Plan) that became the IM4Q measurement program. 

There are two critical features of IM4Q: independent monitoring and the process of “Considerations.” The interview team is independent, which means they are not from ODP, the county, or the individual’s service provider. The independent teams, primarily consisting of people with disabilities, family members, and interested others, monitor the quality of services for people with disabilities. Considerations are a mechanism whereby individuals, their family members, staff or the interview teams (monitors) can make requests for change or improvement to services. Considerations are intended to improve the individual’s quality of life.  The Considerations process is woven into the state’s HCSIS (Home and Community Services Information System). Upon entry into HCSIS, the individual’s supports coordinator is notified and there is a process called “Closing the Loop” that has a specific protocol for addressing the request identified in the Consideration.

The IM4Q is administered at a local level through each of 48 AEs in Pennsylvania.  The AEs select local agencies to conduct the interviews.  Standards for selecting local agencies include being independent of service delivery entities and having consumer and family involvement on governing boards. They must have the participation of individuals receiving supports and families in data collection activities (ODP, 2016).  Each local agency has a coordinator who is responsible for hiring, training, and supervising interviewers.  Also, the local coordinator is responsible for ensuring fidelity in implementing the IM4Q procedures and the ODP’s HCSIS systems correctly.

IM4Q samples for each data collection are randomly selected. A sub-sample of people receiving supports through the PFDS waiver (Person/Family Directed Support) was also included in addition to those who are a part of the NCI sample and living in residential settings. The local programs received lists from ODP with randomly selected individuals to be interviewed.  They are responsible for assigning an interview team. A subsample from each state-operated center (public ICFs/ID) is interviewed each year.  In addition, while the sampling each year is random, many individuals are selected for interviews every other year or every three years.  Due to the longevity of the IM4Q program, this allows for the ability to follow individual outcomes over time and to track the effects of state policy changes. Participation in the interview is voluntary and is scheduled with the individual.

Institute on Disabilities, Institute on Disabilities, Temple University’s  Institute on Disabilities provides training and technical assistance to local IM4Q programs about the Essential Data Elements (EDE) survey.  The EDE includes questions from the National Core Indicators Survey as well as information related to the IM4Q’s Considerations. It consists of items in the following areas:  the Pre-survey which includes contact information; Pre-survey addendum; Satisfaction; Dignity, Respect, and Rights; Choice and Control; Relationships; Inclusion; Monitor Impressions; Major Concerns; and, the Family/Friend/Guardian Survey.  The Institute on Disabilities receives the data to analyze and creates reports.   Reports include easy to read reports for self-advocates and others are prepared and available to individuals through ODP’s “My ODP” website.  Also, in 2019, provider level reports were introduced for providers that had ten more individuals receiving supports who were interviewed as part of the IM4Q process.

Training

Training of monitors (what interviewers are called in the IM4Q process) happens at the local level.  Each local organization is charged with training the monitors that they hire.  Regional training about the EDE is provided by Institute on Disabilities, Institute on Disabilities, Temple University’s IM4Q’s technical assistants.  Monitors are provided with a training manual.  Training topics include the survey itself, survey/interviewing procedures and protocol, review of all of the survey questions, and the intention behind them, disability etiquette, ethics, and understanding of Pennsylvania’s service system. Mock interviews are conducted as part of the training process. Once monitors finish training, they shadow experienced monitors before conducting interviews.  The local IM4Q coordinator will also shadow monitor teams to ensure that interviews are being administered correctly.  In addition to the in-person training, monitors have access to the College of Direct Supports for additional training about disabilities and both ODP and Temple will have online training for IM4Q coordinators and monitors. Annual statewide training conferences are held to exchange ideas, share new developments, and learn different training strategies.

Interviewers and local agency coordinators did have some suggestions for improvement for the training program.  These include different training opportunities for experienced monitors to keep training fresh. 

Technical Assistance

Technical assistance is offered at multiple levels depending on the technical assistance question. Institute on Disabilities, Institute on Disabilities, Temple University’s IOD staff serve as technical advisors for the project. They are available year-round and provide remote as well as face-to-face assistance. They have developed a repeatable training program for all stakeholders and have a variety of materials to assist in program planning and implementation. All four regions of the state have quarterly meetings with the technical advisors, and there is an annual IM4Q statewide training.  Local AE staff serve as technical assistance staff for local agencies. State ODP staff from each of the 4 regions are also available to provide technical assistance related to IM4Q and entering data into the HCSIS, as well as from the ODP central office IM4Q specific staff.