The proof of the pudding is in the pudding

A good cookbook not only lists ingredients and quantities but also goes into the detail of when to add them and describes minutely how they should be combined.

Why? Simply because it makes a difference to the outcome; in the case of good eating, to the taste and texture of the meal. Much the same goes for evidence-based programs: the detail of the implementation matters.

The reasons why programs fail in the real world often have more to do with how they were implemented than with any inherent weakness in the recipe. Meta-analysis, the study of studies on a specific topic, shows that implementation is the most important feature where the success or otherwise of program outcomes is concerned.

Clinical psychologist Joseph Durlak [1] at Loyola University, Chicago, and his colleagues found as much when they examined over 500 research studies of evidence-based programs. There is credible and extensive empirical evidence that the level of implementation affects the program outcomes, they report.

They also found that positive results were achieved when an implementation level of between 60-80% was reached. Just as well, since no study scored near-perfect implementation by their measures.

Influential factors were found in every aspect of the implementation process. At the community level, for example, political support and adequate funding were important. As predictably perhaps, implementers who acknowledged the need for a particular program and believed it worked were more likely to do the job well.

In the same vein, programs that fitted the implementing organizations mission and values and also possessed some inbuilt flexibility were more likely to be implemented faithfully.

Durlak's verdict on fidelity confirms previous findings about the balancing importance of adaptability. It is true, he argues, that higher levels of fidelity are directly linked to better program outcomes, but since no program is implemented with 100% fidelity some scope for intentional adaptation needs to be allowed.

The bottom line is that adaptation should not tamper with the core components of the program. Hence the importance of monitoring. Efficient monitoring was consistently linked with better outcomes; where it occurred effect sizes were as much as three times larger.

So measuring the effectiveness of implementation and establishing a feedback loop with the goal of improving implementation quality is becoming an increasingly familiar requirement. will increase the likelihood of achieving impacts.

[See also: Adding the nerve to working together [2]

Durlak J A, and DuPre E P (2008) Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting Implementation American Journal of Community Psychology, 41, pp 327-350.