Knowing what works isn't altogether working

The creator of the standard-setting Blueprints program told his Colorado conference audience, this week, that after a decade of meticulous evidence gathering, it was sobering that the US should have spent over $40bn on largely useless drug interventions but still have recoiled from the prospect of investing $550m in putting Life Skills Training into every school.

Professor Del Elliott was reviewing the major obstacles to the broader implementation of evidence based practice.

At the start of his career nothing worked, he said. Now it was clear that quite a lot worked – Life Skills Training was a prime example – but, just as surely, as much if not more in the policy arena did not.

His starting point was continuing confusion over what counted as a proven model. Why did we say that PATHS was a model program but FAST Track merely a promising one?

Blueprints was one of eight lists of proven programs supported by the US Federal government, he explained. Only one program, Life Skills Training (LST), was counted in the most effective category by seven of the eight. None hit the top notch on every list.

An expert group on standards for certifying programs as effective had been established by the US government to look at the consistency issue. Their findings made uncomfortable reading for most program providers.

A model program was one where evidence of impact had emerged from a randomized controlled trial, where the effect had been sustained for at least a year after the intervention, where there was at least one replication of the evaluation by a team independent from the originators, and where the study designs dealt with threats to the internal validity of the evaluations.

A model program had to meet all four criteria. Between 10 and 25 per cent on current Federal lists qualified.

He conceded that it was a high standard. But he argued that attaining this threshold ought to be a precondition of taking a program to scale.

He went on to criticize the woeful lack of interest in questions of fidelity. Programs did not work unless they were properly implemented, he said, but most evaluations still focused on outcomes, ignoring the vital contribution made by implementation quality.

Other aspects of program evaluation troubled the Blueprints expert panel as it tried to distinguish between effective and ineffective programs. Their internal validity was often unclear. Problems of attrition, effect size and inconsistency in measurement made judgments about impact difficult. External validity was equally important, as was some indication that the program was relevant beyond the context in which it was evaluated.

All this had convinced him, he said, of the need for a single standard of reporting experimental evaluations.

It was also important to encourage investigators to report negative results. Acknowledging what didn't work was just as valuable as knowing what did.

Elliott highlighted what he called the new research frontier of dissemination and implementation of evidence based programs. Too little was known about the processes required to take effective programs to scale.

One task was to let people know that evidence based programs existed. Given the low rate of take-up and policy makers’ willingness to invest in potentially harmful alternatives, it seemed safe to conclude that most people did not know.

When they did find out, the job was to get the right fit between intervention and need. Too often sites simply assumed that they should take whatever happened to be “top” of the various lists.

Site preparation was another challenge. Many failures occurred because a site had not been properly prepared for change: they lacked a local champion or the necessary administrative support or organizational stability. In any community it was vital to establish credibility for the work.

Training and technical assistance during the change process were just as crucial. Technical assistance tended to decline with time; there was a tendency to react to problems rather than to anticipate them.

Striking a good balance between program fidelity and skillful adaptation to suit local conditions was another ever-present challenge. It was only natural that teachers in Sweden, for example, should want to adapt an intervention designed in Tennessee. Adaptation was permissible as long as it did not compromise the internal logic of a program, but the need for it was often over-estimated. More often, adequate time was needed for the community doing the implementation simply to get the measure of what was being proposed, to understand why and with what purpose.

Elliott closed his presentation on Prevention Action's home turf by reflecting on the need for better dissemination of evidence based programs and practice.

The evidence existed, but the take up was poor, he said. About two-thirds of US schools ran substance abuse programs, but less than a third used proven models. In the aftermath of the Columbine High School massacre just a few miles from the Denver conference, which claimed 15 lives in April 1999, the focus of the policy debate was not the prevention of violence but the quality of the police response and victim support.

• See also
the National Registry of Evidence-based Programs and Practices [1]
the US Department of Education's Safe and Drug-Free Schools Program (SDFS) [2]
The Office of Juvenile Justice and Delinquency Prevention's Model Programs Guide (MPG) [3]
Preventing Mental Disorders in School-Age Children [4]