Would any research scientist be so wantonly unscientific as to make a dodgy claim about the effectiveness of a program – considering the damage likely to be done to the cause of knowledge-based practice, worse the potential damage to children and families?

The truth is that shortcomings and disparities in the methods used to gauge effectiveness mean that unfounded claims of service effectiveness abound.

They aren’t quite of the same blatant order as Nestlé, for example, making the vacuous claim that its Ski Activ8 yogurt contains vitamins proven to optimize the release of energy from your diet (combined with healthy diet and lifestyle) or the Champneys (of health spa fame) claim that their (vinegar) foot patches extracts toxins from the body while you sleep.

But a press release I picked up at a London conference recently proudly proclaimed that a certain initiative had “succeeded in reducing truancy, exclusions and offending” when it was plain from the evaluation reports on the display stand that despite much good work, nothing on offer had the rigor needed to support such a statement with confidence.

So the audacity and intent to deceive are perhaps not replicated in children’s services, but the effect is the same: to mislead through the use of pseudo-science.

Notwithstanding recent interest in this area and with the exception of several popular “proven models”, it’s still the case that services are not really designed; they emerge. They are rarely the product of systematic development and testing underpinned by scientific research. Unless they are clinicians as well, researchers have tended not to take much interest in how services are formulated.

As such, services can be as much subject to fads as the lifestyle industries. In my slot at the London conference I used a painting of two canoeists to remind the audience of the fashion a few years ago for viewing outward-bound courses as the solution for youth disaffection. I asked how such an intervention could conceivably interrupt the chain of risk that leads from overcrowded housing to anti-social behavior via inconsistent parenting from a depressed mother. The answer, of course, is that it cannot; the suggestion is plainly ludicrous.

Homeopathic remedies provide a useful analogy. They are claimed to cure everything from eczema to HIV Aids. They are formulated by the repeated dilution of an ingredient until there is not a single molecule of it left in the final dose (the technical term is ‘30C’ – meaning that the original substance has been diluted by 1 drop in 100, 30 times). It is impossible reliably to distinguish these solutions from water. They contain no active ingredient, and as such there is no logical reason why they should achieve the ends they seek. The twist, of course, is that some people swear by them; they take a homeopathic pill and they feel better. Unfortunately, sugar pills and saltwater injections produce a similar experience. It’s called the placebo effect.

There has been important progress in children’s services in recent years. Today outcomes are embraced and the need to evaluate services is assumed. People talk the talk. My point about the canoeists is less likely than before to be opposed: it is increasingly accepted that the services we develop must logically have the capacity to address the problem in question and should be exposed to an RCT evaluation.

However, not everyone understands the implications of what they are saying. They don’t realize that they are not walking the walk. Before they can do that certain faltering first steps are necessary.

First, researchers need to be honest about the value and limits of different scientific methods and not misrepresent methods they do not favor as part of some pointless academic turf war. Arguing over whether a spade is better than a toothbrush is absurd; it depends if you want to dig the garden or brush your teeth. In the same way, RCTs are good for measuring efficacy but not for gauging user-satisfaction.

Second, researchers need to highlight and challenge pseudo-science in its various manifestations and argue for more resources to do research properly. A research tender that landed on my desk recently wanted to know the impact of a service on outcomes in multiple sites (for children, families and communities). It also requested a cost-benefit analysis. There was no indication that the model meets basic evaluability criteria, yet data collection should start immediately and take no more than three months. This is rather like asking NASA to reach the moon with a washing-up liquid bottle.

Third, the research community needs to demonstrate what is possible by disseminating good practice in the design, implementation and evaluation of services. When it’s done well the results can be encouraging for all concerned.

For example, numerous databases of effective services now exist. Producing such services is a challenge, however. How can it be done? The Prevention Research Center in Pennsylvania, US, uses ‘learning communities’ to foster collaborations between prevention scientists, technical assistants and community level staff.

My own work at Dartington Social Research Unit involves bringing several partners together around the same table to follow a structured service design methodology. Looking forward there’s a strong case for developing a new career path for service designers with an equal and in-depth knowledge of research and practice.

We’re convinced by the value of this democratic but carefully managed design process and have for long argued that each agency should have a position of Chief Service Designer with the same prestige enjoyed by Apple’s chief designer, Jonathan Ive.

If I were a child what would I do given the choice – and understanding necessary to make that choice – between much of what is on offer and something rigorously designed and tested? Improving children’s services might not be rocket science, but maybe it is more rocket science than we have tended to think.

[To read more about my examples of pseudoscience see Sense About Science: There Goes the Science Bit]