What have the following got in common: the Tacoma Bridge, the Challenger space shuttle and the collapse of the World Trade Center? Sadly, they are all disasters that cost human life, and the disasters can, in part, be attributed to poor design. Each of these examples is cited by Henry Petroski, a well-regarded civil engineer and professor of history, who argues that failure is, in fact, an integral part of the design process and that, time and again, success has been built on the back of failure rather than through the easy imitation of success.

Failure is, of course, an inevitable fact of life. In every sphere – personal, professional, business, academic – there will be failure. What is not inevitable, however, is how that failure will be treated. Does it become an opportunity to learn, or something to hide, like an embarrassing family secret?

Failure may be too harsh a term in the context of research, but it is not unusual for studies, particularly those using robust methods, to find that a program or intervention does not produced the results that were hoped for.

What happens in these situations and how do we treat research that fails to generate positive findings? Although some such papers get “filed in the bottom drawer”, this was not the case for Stephen Scott and his colleagues’ excellent study of a parenting intervention program which took place in one of the most deprived communities in England.

On the positive side, the program succeeded in improving the parent-child relationship. No mean feat in and of itself. Unfortunately, the program failed to make an impact upon the two main child outcomes – behavior and reading ability – it was designed to address. While it would be easy to dismiss the program and consign it to the “doesn’t work” category, the report reveals many rich lessons, not just for this particular program, but for other programs more generally.

The program in question is PALS, a careful blend, with some adaptations, of the Incredible Years parenting program and SPOKES (a literacy program). Robust evidence exists to show the effectiveness of both programs. The research team set out to look at what happens to PALS when it is put into actual practice by staff. The research team’s hunch is that there may be “considerable clinical, public health and financial benefits to society from disseminating the use of suitable parenting programs on a large scale”.

The study was an intention-to-treat, randomized controlled trial – the gold standard for evaluating impact. Of the 672 five and six-year-olds attending four schools in an ethnically diverse, inner-city area, 174 were selected. Half were allocated to the intervention group and half to the control. Those in the control were offered services as usual. Parent-child relationship quality, child behavior and child reading were measured before the program and after one year.

While the program only managed to affect parent-child relationship quality, why did it not improve other outcomes? There are several possible explanations.

First, and most obviously, was there, in fact, a reason to believe that the program should have worked? This appears to be the case; in robust studies, both programs - Incredible Years and SPOKES - had been shown to work.

Second, was there a failure by the staff who were operating the program? As part of the effort to give the program a practical, realistic setting, it was not delivered by professionally trained staff with – for instance, nurses, clinical psychologists, and family therapists - but rather by staff who are more typical of the children’s services workforce. In this case, graduates with experience of working with families. But a problem with the staff doesn’t seem to be a plausible explanation. When observed, these staff delivered the program well.

Third, was the program reaching the right families? The research team deliberately set out to be inclusive and to avoid stigmatizing families. All children were screened for behavior difficulties with approximately one-quarter scoring high. The program was offered to a mix of parents of children with and without behavioral difficulties. The parents of children with behavioral difficulties were as likely as the parents of children without difficulties to accept the offer of the program.

The research team also built strong relationships with the schools to gain the trust of families in the school community. Great efforts were made to engage with so-called hard-to-reach families. Even then, however, one-third of families who agreed to take part in the research, and were in the intervention group, didn’t make it to a single session. This may, in part, be due to the fact that the program was offered during working hours, thus making it hard for working parents, or those with caring responsibilities and partners at work, to attend.

Fourth, was the program suitable for an ethnically diverse population? There were no differences between the white British population and ethnic minority groups on either take up of the program, attendance or satisfaction.

Finally, was the program delivered properly? There are a couple of ways to think about delivery with fidelity. First, are participants getting the amount of sessions that are considered to be optimal? Out of a possible 18 sessions, the majority attended less that half. Parents may simply not have had enough sessions to change their children’s behavior and reading.

Second, is the program offered as designed? As a result of the arrangements for training, accreditation and supervision of the facilitators, the program was offered at the correct frequency (weekly) and with the correct materials. Having said that, to ease delivery, the SPOKE program had been shortened and it could have been the reduced number of sessions that led to a reduced effect.

So, there are a number of lessons here.

First, it might not be the case that parents are hard to reach, rather that services are hard to reach. Families appear to want help but attending a group offered during working hours is simply not feasible for many families. It should be incumbent upon those delivering services to make their services accessible.

Second, while being inclusive probably helps to reduce stigma, offering an expensive intervention (estimated to be at least £1350/$2100 per child) to families who are unlikely to see much benefit (if your child is doing well, there’s not much room for improvement) might not be the best use of limited resources.

Third, we probably do not need to design different programs for different ethnic groups. It seems that when it comes to parenting, there are more similarities than differences across cultures.

Fourth, these types of interventions can be delivered by relatively inexperienced graduates without professional training, as long as they have the basic competencies to work with families, are trained and accredited to deliver a program, and receive supervision of a sufficiently good quality.

If we are to learn from our failures, business as usual will really need to change to make the routine implementation of the right program, to the right families, by the right staff an every day reality.

References:

Petroski, H (2008) Success through Failure:
 The Paradox of Design
, Princetown University Press

Scott, S., O’Connor, T.G., Futh, A., Matias, C., Price, J. and Doolan, M. (forthcoming) Impact of a parenting program in a high-risk, multi-ethnic community: The PALS trial, Journal of Child Psychology and Psychiatry.

Scott, S., Spender, Q., Doolan, M., Jacobs, B. and Aspland, H. (2001). Multicentre controlled trail of parenting groups for child antisocial behavior. BMJ, 323, 194-197.

Scott, S., Sylva, K., Doolan, M., Price, J., Jacobs, B., Crook, C. et al. (2010) Randomized controlled trial of parent groups for child antisocial behavior targeting multiple risk factors: the SPOKES project. Journal of Child Psychology and Psychiatry 51, 48-57.

Links:
see http://www.preventionaction.org/what-works/london-school-trial-speaks-we... for a description of SPOKES