Today we visited the Washington State Institute for Public Policy and heard from Steve Aos, Robert Barnoski and Stephanie Lee about their work for the state legislature on the cost-benefit of different programs.
The group’s discussion afterwards focused on the meaning of ‘evidence-based’, the implications of evidence that a program doesn’t work, and how best to get politicians engaged with the evidence.
“I was stunned to see the negative effects of programs that we happily roll out, such as Family Group Conferences”, commented Pauline Curran, representing Together 4 All, Northern Ireland. “These originated in the New Zealand justice system and now are used across the UK in childcare, education, and so on. I’ll now be looking in precise detail at everything we do”.
She added, “I’ve been trained in FGCs but I’ve not delivered it. That was eight years ago, so what’s the probability of me adhering to the model? It would be instinct not knowledge.”
Paul Morgan, also representing Together 4 All, said that the evidence made him question whether government and agencies in Northern Ireland are focusing on the right things. “The government has set a target with FGCs to have so many families going through each year. But no-one is looking at why families didn’t engage or at whether the service produces positive outcomes for family.”
Questions were raised, though, about the validity of the evidence presented by Aos and his colleagues on FGCs. “I’d like to know more about the fit between the indicators of impact used in the study and the intended program impact, since a primary goal of FGCs is family engagement”, observed Judith Wildfire, Senior Consultant for the Family to Family program with the Annie E. Casey Foundation.
“There are more characteristics of good science than RCTs” she added. “For example, the data on abuse and neglect, which focused on reported incidents, was problematic. There could have been more reports owing to the FGCs. You need measures of actual abuse and neglect.”
Michael Little, from The Social Research Unit at Dartington, suggested a way forward. “With any of these programs we can say ‘I’m not convinced, I want to find out if it has an impact’ and then evaluate it. That’s within the gift of all of us.
“RCTs are the best way of determining impact, and we should rely more on systematic reviews. Too often in UK with evaluations there is a sense that you buy the opinion of professor. With RCT it’s the method that tells you if the program has an impact. FGCs are a good idea that doesn’t work.
“RCTs offer an incredibly valuable methodology and in Europe we’re not doing them enough. When we started working in Ireland five years ago there had been just two RCTs in children’s services on the whole island.”
The discussion moved on to consider the connections between research and policy. “What was really positive and strong in today’s presentations was the influence of the evidence on legislators”, commented Julie Anderson, from Norfolk County Council Children's Services. “In the UK it seems that a civil servant comes up with an idea and those of us in children’s services are told to implement it. We’re lucky if they talk to practitioners at all.”
Joyce Moseley, Chief Executive of Catch 22, shared this sense of despair. “I’ve been in so many meetings with ministers and senior civil servants, and they do look at the evidence. But then politics intervenes. People say ‘There’s no way that’s getting through’ and it gets pushed aside.”
Michael Little suggested that one reason for this was that researchers weren’t adept at sharing evidence with policy makers. “One of things we have seen here is programs that have evidence base. But most of those that that have been implemented widely also have great communicators whose selling skills are key to getting the program to go to scale. The flipside is that those people can become evangelical about their own program, even competitive.”
The discussion moved on to talking about getting politicians engaged with research. Suitably, the discussion was kicked off by Graham Allen, MP for Nottingham North. “The joy of hearing from Steve Aos and his work with the Washington State legislature is seeing people from think tanks and science interacting with politicians. We don’t see it in the UK where there are problems to do with the separation of powers, how seriously civil servants take evidence, focusing on the flavour of the month.
“If money is limited and you continue with a bad program because it brings money into your public service organization, you are not leaving space for a good program that is outside the door that will be far more effective. We keep stuff going for decades when it’s not worth it, yet Family Nurse Partnership has taken 26 years to break into the UK.”
Barton Parrot, from the Center for Applied Research, identified one factor contributing to this. “All presentations have been perceived as non-political. You need multiple roles – people who are evangelical but also people who are dispassionate.”
Jane Lewis, Director of Research at the National Children’s Bureau, added: “It’s a real advantage that they’ve been able to talk the talk consistently. The Washington State legislature doesn’t have to engage with one group of academics on early years, another group on juvenile crime, and so on, and you can see how that credibility would build up after interaction after interaction. It’s harder if politicians have to speak with different people on different issues.”
Nick Axford, from Dartington, expressed frustration about politicians who preach evidence-based practice but in the next breath heap acclaim on a program like SEAL (Social and Emotional Aspects of Learning), which is widely rolled out in the UK despite not being proven to work. “In Birmingham we are trying to get schools to implement the PATHS program (Promoting Alternative Thinking Strategies), one of the 11 programs on the Blueprints list, but it’s been proving difficult to get schools to sign up for it because they say ‘We’re doing SEAL’.”
Jane Lewis put this in a larger context: “We have a long and unhappy tradition in the UK of seeing something that has worked in US and adopting some bits of it but not others. Intuitively this seems like a mature, pragmatic approach. But what I’ve heard this week is ‘Stop doing it! Adapt by all means but do it in a conscious, evaluated way.' In the UK our tradition is to encourage a hybrid approach.”
Michel Little summed up. “First, what do we mean by ‘evidence-based’? We need clear criteria. Second, the programs we like don’t necessarily work. Third, decommissioning. There is so much excitement in doing something extra but the hardest bit of our work is getting people to stop doing things that are probably ineffective. Lastly, we’ve seen evidence that non-adhering therapists implementing an evidence-based program make children’s lives worse, not better. This is incredibly powerful. If that’s happening with trained psychologists who have been trained in FFT (Functional Family Therapy), what does that mean for what happens in ‘normal’ services – health, teaching and so on?”