Governments and local authorities are increasingly likely to stipulate that their services for children should be “evidence-based”; but what passes for “evidence-based” varies enormously from flagrant misuse of the term to a bold affirmation of good science.  [See the Prevention Action special edition about Standards of Evidence.]

Rising evidential standards have bestowed high status on some programs, with the result that the handful able to claim multiple independent experimental evaluations are often categorized as “model” or “proven” by the independent organizations surveying the program landscape.

The push for higher evidential standards has led, in turn, to an increasing number of “brand-names” among programs for children and families. Their off-the-shelf ingredients include varying amounts of commercially available materials, training and technical assistance. 

On the positive side, the “market leaders” are well specified, logical and ready for the oven of local implementation. The evidence suggests in pretty certain terms that they provide an effective way to improve children’s outcomes. But, on the down-side, if customers always choose the prominently-branded and highly visible products, many innovative, “organically-grown” programs or interventions may be left on the shelf.

Mark Lipsey, Director of the Peabody Research Institute at Vanderbild University, Tennessee, is among those who argue that there is more to an “evidence base” worthy of the name than simply the accumulation of tried and tested brands.

He told this year’s Blueprints conference in San Antonio: “I’ve got to admit that I struggle somewhat with the term ‘evidence-based program’. Why? Because there’s more than one way of defining a service”. 

Lipsey reasoned that by limiting itself to the small world of programs substantiated by high standards of evidence, prevention science was in danger of stifling innovation. The definition of a service should not be confined to tightly packaged programs; it embraced a range of generic intervention types – he called them “specific operating procedures” – as well as numerous policies and general practices. 

What gives him the authority to challenge the might of the emerging status quo? 

His co-authored book Evaluation: A systematic approach is regarded by many as being a standard work on evaluation methods.  And, as the holder of the Mosteller Award from the Campbell Collaboration, he is recognised as a world expert on systematic reviews and meta-analyses. 

In Lipsey’s eyes, the range and power of meta analysis provide the key to a more dynamic evidence base. Rather than focusing on the few experimental evaluations of any specific branded program, meta-analyses combine tens or hundreds of experimental studies of a broad intervention type, such as restorative justice, antisocial behavior or cognitive behavioral therapy, to extract a clear picture of their effectiveness – and variability. 

Lipsey told delegates in San Antonio that some studies almost invariably uncovered negative effects. No matter how compelling one intervention approach might prove for a specific problem there were invariably a few conditions in which it would prove ineffective or even harmful. 

He and his colleagues used such negative but all the same valuable findings to learn lessons for the future. The negative culprits were usually poorly implemented interventions or those where the fit between need and service was imperfect. 

Meta-analysis did not undermine the credibility of branded programs, but it was able to investigate ideas outside their orbit: many “unbranded” interventions were just as effective in certain circumstances.

A Blueprints advisory board member himself, Lipsey does not contest the evidence or value of branded programs. He maintains only that by focusing on the often-limited evidence-base supporting individual branded programs we may neglect a wider world of useful possibility.

Meta-analysis creates an avenue for the design and refinement of a more comprehensive arsenal of effective interventions for children. 

See: Rossi P H, Lipsey M W, Freeman H E (2003) Evaluation: A Systematic Approach (7th Edition). Sage, Thousand Oaks.