Considering how heavy the influx of programs bearing the “evidence-based” tag, and how widespread the confusion among policy makers and practitioners about which ones to pick, US prevention scientists had an unexpectedly easy time coming to an agreement about setting standards.

Chair of the Society for Prevention Research standards committee Professor Brian Flay reflects that, “most were agreed upon right away without much dispute - aspects such as reliable scientific measurement, clear descriptions of samples, robust control groups, ideally delineated by randomization, and so on”. 

As Flay and the rest of his committee saw it, all these were requirements that no methodologist worth his salt would dispute. But in reality they are still far from standard practice. 

There were a few far thornier issues. In the end, the final draft settled for two standard levels – the “required” and the “desirable,” the first fundamental to any understanding of whether an intervention works, the second a source of greater confidence and authority.

“Many of the desirable standards would greatly benefit our understanding of whether or not an intervention works, and also why it does,” Brian Flay explains. 

The majority require closer tracking of the steps that connect program components with predicted benefits. For example, a parenting program may have an impact on a child’s schools grades as a result of improving their behavior. At present too many evaluations focus on the start and finish, neglecting the process in between. 

Nevertheless, he believes the field is moving in the right direction. Were the standards to be redrafted today, he says, certain of the original “desirables” could be upgraded to “required" status. For example, there is now sufficient consensus that cost-benefit analysis should be an aspect of normal procedure.

Along with the recent Institute of Medicine report on prevention, latest developments indicate that this young branch of science will continue to evolve. [See: National Academies outline pathway to prevention].

Applying the standards in the real world

Applying the standards in the real world has proved rather more problematic and there is one particularly glaring difficulty - the “desirable” standard for independent replication. 

Scientists agree that a cornerstone of evaluation in all fields is replication of results. One set of positive results, no matter how robust, may be due to chance or other factors not controlled for. 

Most evaluations and replication studies are carried out by the team who developed the intervention. Very few have their findings replicated by a completely independent panel of researchers. 

“It's a hard sell for developers,” explains Flay, “There is little perceived gain; they already feel confident their intervention works so why subject it to further scrutiny which may undermine it?”. 

The issue has proved so thorny that the SPR has made it the business of another task force. 

• Tomorrow Prevention Action talks to Jon Baron, President of the Coalition for Evidence-Based Policy and a vociferous advocate of the standards, about how in the US they have been used to lever unprecedented investment in evidence-based interventions.