The law of diminishing returns lies at the heart of economics. It is the reason why even the smartest entrepreneur cannot dominate the world forever. We might start off loving a new product like the Windows operating system but each new unit that is produced holds a little less value. At some point, the returns evaporate, other products, like the Ipod and Iphone, gain the competitive edge, and Apple grows larger than Microsoft. Eventually, another competitor will rise to challenge Apple’s supremacy.

Exactly the same problem occurs with evidence-based programs. The first unit—a parenting class or a therapy session—has a big impact on a child’s well being. The second unit—the next parenting class—helps a little less. The next less again until in the end there comes a point where even the best evidence-based program is no longer benefiting the child.

When does that point arrive? Recent research by Brandon Walsh, Christopher Sullivan and David Olds reported in the current edition of Prevention Science sheds light on the issue.

Walsh and his team looked at the extent to which economists and other researchers ‘discount’ published results to try to anticipate what will happen in the real world. For example, Steve Aos, widely renowned econometrician at the Washington State Institute of Public Policy, halves the effect sizes of even the most robust evaluation if the program developer has been involved. In the real world, the thinking goes, the program developer cannot support every application of the model, so it is reasonable to assume that the impact will be less than in the artificial conditions of the evaluation.

Peter Greenwood and colleagues, for example, applied a penalty of 40 percent when estimating the cost benefits of prevention and early intervention for crime reduction activity in California in the mid 1990s. Greenwood accepted that there were effective ways of reducing crime, via home visiting, day care, parent training and giving incentives for students to stay in school. But he also assumed that when these interventions went to scale they would not always get to the children who could most benefit and that the long-term impacts would diminish.

Once the discount rate of 40 percent was applied, Greenwood concluded that providing incentives to students to stay in school was the most cost-beneficial option he considered.

Later in the 90s, Donohue and Sieglman applied a larger penalty of 50 percent to the Perry pre-school program and a family development model out of Syracuse University. The researchers had calculated that investment in prisons of between $6 and $8 billion dollars at 1993 prices would produce between a five and 15 percent reduction in the crime rate. What would happen, asked Donohue and Sieglman, if the same money had been invested in the two early intervention programs?

Before the discount for going to scale is applied, the early intervention programs easily win out. Money spent on them would produce a much better return than money spent on prisons. But after the discount rate is applied, reducing for example Perry pre-school’s impact on crime from 40 to 20 percent, there is little to choose, on economic grounds at least, between early intervention and prison.

The ‘go to’ person on discounting is Mark Lipsey. His work is regularly cited by Steve Aos. About a decade ago, Lipsey looked at many studies examining the impact of Cognitive Behavior Therapy (CBT) on criminal recidivism rates. Generally, the results were promising. But there is a significant ‘but’.

Lipsey found that trials using quasi-experimental designs exaggerated the effect between 30 and 50 percent. He also found that an evaluation of CBT in routine practice produced results that were between 40 and 60 percent less encouraging than evaluations in the controlled setting of a university.

A randomized controlled trial of CBT in the real world finds on average an effect size of about 0.04. The less robust quasi-experimental evaluation of CBT in the laboratory produces an effect size of about 0.16.

The intervention is a success. But a policy maker thinking about investing in the program should be calculating for it to impact recidivism rates by about 11 percent, not the 49 percent that might occur if the intervention took place in the lab and the law of diminishing returns could be repealed.

Why is the impact of an evidence-based program reduced as it becomes more ubiquitous? Walsh and his team talk about programs going to children that need them less, the inability of systems to deliver interventions evenly to many hundreds of thousands of people and the loss of program fidelity.

But in truth we do not know. And so Walsh and his team conclude their work with a call for an experiment to provide more reliable estimates of the scale of diminishing returns for evidence-based programs. In difficult economic times, maybe the use of such a study will be recognized.

References

Brandon Welsh, Christopher Sullivan and David Olds, ‘When early crime prevention goes to scale: a new look at the evidence’, Prevention Science, 11, 2, 115-125