It's bad science if it's just plain dull

No doubt about it, create the right ground conditions and prevention programs will reduce teenage smoking or help children to make friends more easily and improve their academic attainment.

The fact that these modest advances have yet to have widespread impact is largely due to the slow progress toward getting proven approaches into mainstream practice and policy. 

The barriers are subtle but formidable, which is why, in formulating their standards of evidence, the Society for Prevention Research went two steps beyond questions of efficacy.

As well as agreeing measures that would indicate which programs worked in ideal conditions, they also set standards for effectiveness and dissemination.

Effectiveness is the measure of whether a program works under “real world” conditions. SPR standards say it must be demonstrated by at least two rigorous studies. 

So it may be the case that an intervention been shown to work, but only when bolstered by staff and resources not normally available. Test it in a “natural” community setting and it will fail to make any difference.  

The SPR standards also require intervention designers to estimate proper dosage – rather as pharmaceutical companies must do. It is a tall order and the comparison is very far from perfect, but SPR acknowledge that it should nevertheless be possible to indicate how many hours of involvement will be required and which components should be delivered in what order.

Standards for the quality of delivery should also extend to the ready supply of manuals, training and support to get practitioners up to speed. 

The Society also acknowledged that engaging program participants is a key task. Here the premise could not be simpler: a proven counseling intervention for depression is useless if the target teenage population declines to attend sessions. 

Just as in efficacy trials, establishing with certainty that an intervention is the cause of the observed changes, is another essential. But the challenges of doing so in the context of real world effectiveness are much greater.

Proof of causality is generally associated with successful randomized controlled trials, but ethical, practical and political obstacles stand in their way at the community level. Other robust approaches may be needed.  

As well as finding out whether programs work in the real world, effectiveness studies will also help to find out who they work for. With this in mind, the SPR standards of evidence require large, carefully defined samples. 

The Olweus bullying program, for example, has been highly successful in Scandinavia, but recent studies suggest that it does not have such an impressive impact in the US (see: Bullying evidence given theoretical going over [1]). 

For this reason, effective interventions must also collect data on participation. Knowing exactly who is taking part - and who isn’t - can help program developers decide whether a program needs to be adjusted for harder-to-reach groups.  

For the benefit of policy makers and those whose opinions may need to be swayed, the potential impact of programs must be explained in everyday terms. SPR standards demand that evidence emphasizes practical value in terms of easily-interpreted figures, such as effect sizes or cost-benefits. 

Roll out, roll out

Proving something works in the real world is still a far cry from actually achieving widespread impact. This is where the standards for dissemination come in. 

According to the SPR, conforming programs will be ready to be rolled out to multiple sites. The thinking is that they will have reached a level of fitness sufficient to ensure that policy makers adopt them and practitioners are willing and able to use them effectively.

So, the developers must be able to provide manuals, training and technical assistance – and to prove that, as a result, programs can be implemented with fidelity to the original model. Programs that fail this practical test because they are complicated, inadequately documented or plain dull - are not ready for dissemination. 

To maintain delivery to a high standard, program developers must also make monitoring and evaluation tools available so that sites are able to assess fidelity, participants’ responses and eventual outcomes, as well as helping developers to fine tune the implementation process.
 
The authors of the SPR guidelines lament the lack of research dedicated to dissemination. Very few interventions match up to their admittedly exacting, standards. The next generation of preventionists will have their work cut out.  

See: Flay B R, Biglan A, Boruch R F, Castro F G, Gottfredson D, Kellam S, Moscicki E K, Schinke S, Valentine J C and Peter J I(2005), “Standards of Evidence: Criteria for Efficacy, Effectiveness and Dissemination”, Prevention Science, 6, 3, pp.151-175