Only 11 programs in the category of children’s services – anywhere in the world – have been proved effective by randomized controlled trial, and, even for those, the benefits are modest, the director of one of the UK’s few practice-oriented social research units will tell York conference delegates tomorrow.

Michael Little from Dartington Social Research Unit will argue that if society expects children’s services to measure up to rigorous evaluation, they must be more systematically designed.

He will suggest a number of possible explanations for the dearth of randomized controlled trials among the plethora of interventions designed for children and families. They include obstacles related to ethics, confusion between consumer satisfaction and evidence of effectiveness, and lack of professional training in research methods and evidence-based practice.

All of this means that many of the standard programs delivered in the name of improving outcomes are likely to be ineffective. Some may even be harmful.

He will argue that coalitions between prevention scientists and the communities for whom programs are intended should be encouraged to produce more rigorous designs with inbuilt evaluation methods – providing that the approach is based on a formal “operating system”.

Operating systems describe a way of managing relationships between stakeholders such as scientists, policy makers, community leaders and families. The term is borrowed from the US prevention scientist David Hawkins who regards it as analogous to Windows Vista in the language of the PC.

By the same reckoning, effective programs such as Nurse Family Partnership and The Incredible Years are equivalent to Microsoft Word or Excel. Examples include Hawkins's own Communities that Care, Results Based Accountability and Common Language.

To explain how adherence to an operating system can lead to more rigorous service design, Michael Little will present examples from activity in Ireland funded partly by The Atlantic Philanthropies and partly by government.

Communities were invited to apply for funding for new services. Led by a representative or leaders of voluntary organizations, the communities had to be economically disadvantaged, to have a child population of between 3,000 and 7,000 – and to be willing to use an operating system.

The majority of applicants chose the Common Language method developed by Dartington Social Research Unit. It has four stages beginning in round table discussion between representatives from children’s services, private and voluntary organizations, children, families, policy makers and others.

The first step is to arrive at a broad strategy about the outcomes the community wants to achieve. Second is the design of specific interventions; third is the preparation of a program manual for each intervention. The fourth step is evaluation - although Little will argue that evaluation is integral to the whole cycle of work because Common Language requires participants to treat all propositions (and designs) as hypotheses and their implementation as a scientific experiment.

The process is informed throughout by the results of epidemiological data on the target population and the views of children and families.

Use of local data means that expectations about what can be achieved are realistic and designers have something to measure their success against. And, by testing the resulting “theory of change”, Little maintains it becomes easier to explain why a program has had the desired effect.

The process is also specific about the target group, the level of demand for the service and potential supply. It allows researchers to calculate important statistics, such as power effects and required sample size – both necessary in the construction of an RCT.

The long-term impact of the the Irish initiative on children’s health and development has yet to emerge and the information will depend on the quality of the services that have been designed and of the evaluation studies they specify.

At present there are four community engagement sites and they have the potential to result in between one and six experimental evaluations. Taken as a whole the work in Ireland has resulted in at least eight new RCTs in children’s services.