There’s no point improving a system of services if the services themselves aren’t very good. This was the lesson Len Bickman and his Vanderbilt University team took from their seminal 1995 evaluation of the Fort Bragg Demonstration Project. That project, profiled yesterday in Prevention Action, theorized that an improved continuum of care for children’s mental health would improve outcomes. When it didn’t, Bickman had to reevaluate and realized that to make a real change, there had to be improvement on the front lines—with the providers and the preventions they offered.

Bickman knew what everyone in the field of evidence-based programs knew, that it is difficult to make the transition from strictly controlled university settings to the community. Often, in adapting programs, service providers change them beyond recognition and their effectiveness is lost. Bickman knew that monitoring programs as they are rolled out is vital. However, few service organizations have the infrastructure to both provide services and evaluate them.

To address this problem, Len Bickman, working through Vanderbilt‘s Center for Evaluation and Program Improvement (CEPI), developed a continuous quality improvement system called Contextualized Feedback Systems (CFS). The system was developed with input from practitioners as well as from scientists and IT specialists. CFS uses the Internet to collect, track, analyze and influence practitioner activity and client outcomes. It was designed to be unobtrusive and fit easily into the provider’s routines.

CFS offers a battery of rigorous and clinically useful measures. These cover processes, such as the young person’s relationship with the practitioner and their motivation to change, as well as clinical outcomes, such as life-satisfaction and functioning. Young people, caregivers and practitioners all provide information to ensure that different perspectives are taken into account.

The data that CFS captures and tabulates has many uses. A mental health clinician can track the progress of a depressed teenager’s symptoms and discuss patterns with his supervisor. A service director can identify problems with caregiver engagement and point her team to helpful training material. A manager can experiment with two evidence-based treatments and evaluate their effects before selecting which one to implement fully.

The information can be analyzed at various levels, from the individual child to the organization’s staff. Feedback reports are provided on-line in a user-friendly format.

Practitioners are encouraged to use the feedback. The system automatically suggests on-line interventions and training modules to address problem areas. These may be supplemented with written manuals and in-person training. Critically, the system aims to support the practitioner’s decision-making without replacing his or her clinical judgment.

The data is flexible. It can also be used to compare practitioners, provider organizations and types of treatment. Children and families have access to the data and can use it to help them select services and providers.

To date, CFS has been used mainly in mental health and general medical practices. But it has also been used in other evidence-based programs. For example, it is now employed as part of a grant from the US National Institutes for Mental Health with Functional Family Therapy (FFT), a treatment program for adolescents with behavioral problems.

Thousands of studies have shown that it’s difficult to improve program outcomes without measuring performance and providing feedback. Len Bickman and his team say that by intervening at the individual level, with the service providers themselves, CFS can trigger organizational change. They are hoping to be able to prove this soon. Check back with Prevention Action for updates.

References

Bickman, L. (2008). A measurement feedback system (MFS) is necessary to improve mental health outcomes. Journal of the American Academy of Child and Adolescent Psychiatry, 47(10), 1114-1119.

Bickman, L., Riemer, M., Breda, C., & Kelley, S. D. (2006). CFIT: A system to provide a continuous quality improvement infrastructure through organizational responsiveness, measurement, training, and feedback. Report on Emotional and Behavioral Disorders in Youth, 6, 86-87, 93-94.

Kelley, S. D., & Bickman, L. (2009). Beyond outcomes monitoring: Measurement feedback systems (MFS) in child and adolescent clinical practice. Current Opinion in Psychiatry, 22(4), 363-368.