How randomized trials are starting to meet the Goldacre standard

Bridging the treacherous divide between good science writing and the newsroom with a smart combination of penetrating analysis and good humor has made The Guardian’s weekly Bad Science [1] column one of the success stories of print journalism in the UK.

Writer Ben Goldacre has won a string of awards for articles that cleverly unpick the relationship between scientific method, politics and society and put pseudoscience, health fads and all other hooey cheerfully to the knife.

The provocativeness of Goldacre's writing has also made it fertile ground for bloggers. His 600-word Saturday column regularly generates several thousand words of reaction before the end of a Guardian weekend – which goes to explain why a recent Bad Science piece on a question not often aired in the UK lifestyle supplements, "Why don't we do randomized controlled trials (RCT's) on social policy?" produced so much stimulating comment. See The Testing of Social Policy [2].

“Like all the best problems,” Goldacre observed “the barriers are institutional and historical: and the objections raised against trials in social policy are exactly the same as those raised in medicine 40 years ago. Judges will say, as doctors once did: we have expertise, we know what works for an individual.” Many social workers will say the same.

“The sad reality is," he reflected, "social policies feel good, like alternative therapies; but, like alternative therapies, most policies don't work. In the field of recidivism, even from uncontrolled studies there aren't many successful interventions.”

All this raises two classes of question - one about the value and effective use of RCTs - the other, possibly by a short leap related to the first, the effectiveness of blogs as a vehicle for informed opinion,

One simple observation on the big puzzle – why policy makers rely so little on randomized controlled trials when they are reputedly so interested in evidenced-based thinking – might be that it’s a symptom of an English system that still sets professional status above knowledge. Paradoxically, considering how science struggles in American society, there is a greater willingness there to rely on RCTs.

Another reason for wariness might be the dismay of not knowing even after expensive quasi-experimental designs and a myriad of ‘soft’ evaluations whether the huge investment in prevention over the last decade, Sure Start included, has paid off.

Lessons from the Irish experience

There may be lessons to be drawn from Ireland. There, a big investment from The Atlantic Philanthropies and a partnership with government will produce upwards of 20 RCTs over the next five years. Prevention Action will report more from Ireland in the coming months; for now it may be helpful to speculate on the conditions needed to make experimental designs more central to the policy process.

A starting point might be franker acknowledgment of how little we know, and how harmful existing policies and services might turn out to be. It may lift the heart to send anti-social young people on adventure holidays, but it is worth considering that such expeditions might be making matters worse. Such negative potential also applies to widely supported programs, Sure Start included.

As one of the respondents to Goldacre's Guardian column noted, the words "I don't know if it works" are seldom heard from politicians and other policy makers. It is true, nonetheless: most of them simply do not know.

As in Ireland, commitment from government and other funders is crucial to the greater use of RCTs. But taking aspects of children's social policy out of the party political process is also important. The prospect of the results of trials becoming available in three years time is unlikely to cut much ice if one political party persuades itself it has a better idea for reducing anti-social behavior than another. But good science demands patience.

A more rigorous scrutiny of ethics would help, too. Those antagonistic to RCTs complain about a situation being engineered in which at least half of the children eligible are refused a service, or at best placed on a waiting-list. Rather less attention is paid to the ethics of providing children with unproven services that have the potential to do them harm. (Remember that those who emigrated children from England to Australasia and Canada did so with huge optimism.)

Arguably the most important element is good service design. One Guardian blogger referred intelligently to 'user-centered design' and to involving the people using the service in the design process. But to many in children's services the very idea of a service design remains alien, to say nothing of the contribution to be made by service users.

In the Irish example, as much effort has been invested in the preliminary design process as in creating the right conditions for the subsequent trial. Indeed, so rigorous is the approach to design and so dependent is it on evidence that the experiments follow quite naturally in the majority of cases.

In other words, it is not a case of philanthropy or the Irish government saying "let there be randomized controlled trials". Rather, they have decided "in the short-term we will invest in well-designed services, and in the long-term we will invest in proven models". With this incentive, local communities and service providers are readily drawn into the rigorous design and evaluation of services.

One of the gratifying aspects of the activity there has been the transformation of local policy makers' and service providers' attitudes. At the outset, most knew nothing about RCTs. Once they knew about them, most were instinctively opposed. Once they got involved, most came around to the idea. And once they were being funded to implement their designs, most become great advocates of the experimental approach.

So Goldacre's Guardian column and its fascinating blog indeed raise important questions. Others in England, for example The Nuffield Foundation and the Journal of Children's Services, will be doing more in the next 12 months to keep the issue live and to turn UK government interest into action.

The goal should not be to view RCTs as the answer to every problem but to make scientific method more central to the policy debate. In other words, to move it from the bloggers' strip nearer to the front page.