A squall blew through several Prevention Action articles last week. David Olds cast doubts over the validity of methods used in the evaluation of the Chicago-Parent Center study and its impact on outcomes. Ogden and Henggeler questioned the findings in a Cochrane review of MST. Researchers in Durham have expressed doubts about the value of Sure Start, but using methods that fall short of the standard that Olds was calling for.

(See: Chicago Child-Parent Center study celebrates coming of age
Durham study casts more doubt on Sure Start
$14m for UK pilots of Multisystemic Therapy Services Inc.)

So what is a policy maker to do? Comparatively few programs have been shown to work using the highest standards of evaluation. The Blueprints for Violence Prevention database lists just 11 'model' programs evaluated by experimental design that have had a sustained impact on children's development and been replicated across several sites.

And even in the US where most of them originate there is comparatively little investment even in these 11 'gold standard' programs.

In his editorial on the Chicago Child-Parent Centers, Olds called for a replication in early years policy and research of the kind of scrutiny that has transformed the treatment of childhood leukemia. The first experiments with treatment began 50 years ago. The pioneers of the experimental approach prevailed, and today 80% to 90% of victims survive.

The analogy is enticing but not entirely convincing. Roughly 4,000 US children a year are afflicted with the type of leukemia Olds describes. Incidence rates for the impairments to development and disorders that concern children's services are much higher. Major depressive disorders, for example, affect more than 3.5 million in the US, but investment in research on children's needs bears no resemblance to the investment in studies of leukemia.

The diagnosis of leukemia is reasonably precise. Where impairments to development are concerned we are often fumbling in the dark by comparison. Progress with childhood leukemia is the result of better treatment, not more effective prevention. And in the case of disease there is seldom disagreement about the need to act – something, alas, that cannot be said of all children's social and psychological problems.

So if leukemia is not the model, what is?

We can all agree that we know too little and that better understanding would not only be good for children, it would also improve society and benefit the economy.

It should be beyond question that there are different standards of evidence and, in the context of knowing whether prevention programs have any impact on child outcomes, bar a few exceptions, experimental methods are the gold standard, Several RCTs summarised in a systematic review provide the most reliable source of knowledge.

That does not mean that other sources of evidence are not useful. What is disappointing about Reynolds's work on the Chicago Child-Parent Centers is not so much the methodology (the weaknesses of which he acknowledges), but the fact that the findings are not more widely used and applied. The Durham study tells us a lot about trends in the cognitive ability of primary school children but comparatively little about the effectiveness of Sure Start, and nothing about children's non-cognitive skills.

A successful RCT will never become the guarantee or prescription that a particular program will or should be widely applied. It will prove effectiveness and indicate that the logic behind an intervention is therefore sound, but the logic, not the particular demonstration of it, is the part that can readily be transferred from place to place and developed.

All the evidence mixed up in last week's squall – experimental and non-experimental – could, with more besides, provide the basis for best policy and practice, and a staging post for finding out more and gradually fashioning successful interventions.

No single program has ever been found to be 100% effective. Even the best have a relatively modest impact on child development, so it is folly to imagine that any one will provide 'the' answer.

As David Olds argues, progress depends on continuous and increasing investment in research and evaluation. And, given our limited knowledge, the work requires a greater humility in the face of the difficulties of helping children and a greater respect for good evidence wherever it is found.

Such a change will involve getting innovation, evaluation and the assimilation of knowledge out of the hands of an elite group of researchers and into routine policy and practice. In other words, let's have children's services agencies innovating, evaluating, sharing knowledge, and regularly updating practice in the light of evidence.

The challenge is not to decide who or what is better. It is getting more people engaged in the rigorous activity of innovation and evaluation.