Policymakers want to spend tight child welfare budgets where they’ll make the most difference for children’s outcomes. In policy and practice alike, “cost-benefit analysis” and “evidence-based interventions” have become mainstream ideas. But are current methods good enough to provide the guidance policymakers need?

In the US, these policy analyses are indeed feasible – but so far, such research has been rare, argues a group of health policy scholars. Given the massive leap forward in economic modeling methods over the last decade, it’s time for researchers to rise to the challenge, says the team of academics, led by Stanford University’s Jeremy Goldhaber-Fiebert.

In a recent review of economic evaluation literature of child welfare interventions, Goldhaber-Fiebert and colleagues claim that the ability of researchers to conduct high-quality cost-effectiveness analyses of child welfare interventions has been greatly enhanced over the last several years. The authors examined 19 reviews and 30 original research articles that evaluated the costs and benefits of interventions.

One factor that has greatly improved the feasibility of cost-benefit analyses of child welfare is the use of computer-based models to simulate the long-term impact of various alternatives in “real world” settings. These computer simulations typically use repeated cycles to examine a variety of scenarios, which are compared to provide recommendations about the best action to take. This allows researchers to take into account real-world populations and policy contexts. These models can also project outcomes and costs further into the future than the original studies were able to do.

As these modeling techniques have become more advanced, they can be used not only to compare individual interventions, but also to consider the costs and benefits of combinations of interventions and services. Such sophisticated analyses are crucial for helping decision-makers predict whether the long-term benefits of evidence-based interventions justify the investment.

But, the review found, there are many gaps in the research. Much of the computer modeling that is currently being done doesn’t use state-of-the-art techniques. Moreover, eight out of 10 reviewed articles that used modeling were written directly for a policy audience and were not peer-reviewed – a step that, the authors suggest, could improve the quality of the assessments.

Methodological guidance: it’s out there
So if researchers want to rise to this challenge and carry out the sophisticated cost-effectiveness and cost-benefit analyses that need to be done, where can they turn for support? Goldhaber-Fiebert and colleagues point to the increase in methodological guidance over the last decade. Several of the articles they reviewed provided methodological descriptions of modeling efforts that can serve as tutorials.

The review identifies articles that were found to be a good resource for anyone who is interested in learning how to do this type of work. These articles may serve as a guide for those interested in developing the evidence base for a program in a way that is relevant to policymakers.

Making research policy-relevant
Using the best data and methods to make child welfare research relevant to policymakers is crucial, the authors argue, because only a small proportion of government spending on child welfare and child protection goes toward evidence-based interventions, despite their potential to improve outcomes and provide a return on investment to the taxpayer. Goldhaber-Fiebert and colleagues identified five major gaps in the research – gaps that, if filled, have the potential to improve the quality of child welfare decision-making.

First, when researchers and practitioners are designing trials of interventions, they can aim to improve their estimates of the costs of interventions – information that is critical for policymakers. Because the costs of interventions vary from site to site, providing details about how costs are calculated can help potential investors to estimate what the costs are likely to be in their area.

Second, estimates of the costs that have been avoided through intervention – that is, of services that won’t be incurred, and conditions that might be avoided – also need improvement. The range of existing cost estimates needs to be extended to enable researchers to improve the accuracy of their predictions. Ideally, researchers and practitioners or service managers would collaborate to identify and summarize the costs of services.

Third, the review noted that studies of interventions were often weak in their measurements of the problems that they were aiming to address, as well as of the quality of life of those receiving the interventions. Using better measures of these important outcomes in economic analyses would allow for an easier and potentially more accurate analysis of the cost-effectiveness and cost-benefit of programs. In general, it is helpful if studies can be designed and the results written with policymakers in mind.

Fourth, among the original research articles, the authors noted that few studies compared ways to work with children who are already in the child protection system. Most of the studies were focused on prevention. More research on the best ways to work with children already in the system, particularly taking into account budgets, would be extremely useful.

Finally, many of the studies did not go through a peer-review process. Although this is because these studies are typically conducted in real-world public policy settings where peer review is not required, peer review would ensure that others working in the same field could provide feedback and potentially improve studies before publication. The authors argue that peer review may actually be even more important when reports are being written directly for policymakers, as they are being relied upon to assist with important decisions about providing services in real settings.

Reference
Goldhaber-Fiebert, J.D., Snowden, L.R., Wulczyn, F., Landsverk, J., & Horwitz, S.M. (2011). Economic evaluation research in the context of Child Welfare policy: A structured literature review and recommendations. Child Abuse & Neglect, 35, 722-740.