Can a systematic review be any better than the work it’s based upon?

by Terje Ogden and Kristine Amlund Hagen
Since the introduction of Multisystemic Therapy (MST) to Norway in 1999, implementation teams have been established in every municipality across the country and about 2,600 adolescents and their families have received treatment.

The program has been evaluated with Norwegian families in two evaluation studies, approximately six months and two years after intake The results showed that, compared to regular services, MST was effective in preventing out-of-home placement and in reducing youth problem behavior.

MST has also been evaluated by the program developer, Scott Henggeler and his colleagues, and by and large these studies also demonstrated significantly better outcomes for families. Less encouraging results come from one study conducted in Canada by Lescheid and Cunningham in 2002.

Despite this, a Cochrane Review by Julia Littell and colleagues has produced negative findings. The conclusions not only contradict the results from the primary studies on which the review is based, they also overturn findings from an earlier meta-analysis conducted in 2004 by Curtis, Ronan, and Borduin, which included six of the eight studies reviewed by Littell. Researchers and practitioners familiar with the program have consequently been left baffled.

On the face of it, Littell and her team included eight MST evaluations in their survey, but it becomes clear on closer reading that the analyses actually relied on between two and five studies, in most cases just three. We believe that these numbers are too small and therefore seriously limit the ability to draw any general conclusions.

Most meta-analyses observe between 50-200 studies. Littell and her team acknowledge this when they say: “Since statistical power is low, we can not conclude that MST is not more effective than other services”. But this recognition of the limitations of their work seems to have been misconstrued as a criticism of the MST program itself.

There is a further double drawback: among the few studies included is unpublished Canadian work by Lescheid and Cunningham which weighs heavily in the meta-analysis because of its relatively large sample of 409 participants.

The characteristics of this investigation are not known because it is as yet unavailable for review by members of the research community, and as long as it lacks peer review its quality must be open to question.

Highest rank order is given to the Canadian study because it conducted an “intention-to-treat” (ITT) analysis. But ITT can underestimate treatment effects and neglect the positive effects on participants who actually completed treatment.

Littell and her team also criticize technical aspects of several MST studies. Their concerns include what they consider to be inconsistent reports of sample sizes, unclear randomization procedures, unequal handling of members of yoked pairs, unstandardized definitions of treatment periods and treatment completion.

Some of these claims are difficult to prove one way or the other, but we believe that the researchers responsible for the primary studies are in a better position to report what is relevant and accurate.
It may be that there has been deviation from the strict norms of conducting randomized controlled trials (RCTs). But total adherence to such standards is unrealistic, especially in effectiveness trials. Whether the results from the MST studies in the review would differ substantially as a result of such conformity is open to question.

Furthermore, the authors of the meta-analysis make a logical slip: when they disapprove of the methods, data, and design of the investigations included in their meta-analysis they simultaneously undermine their own study. How can a systematic review or other meta-analysis be any better than the investigations on which it is based?

The Cochrane Collaboration’s advice to its own reviewers makes the case very well: “If you identify and suspect that important diversity or heterogeneity is present in your review, there are several options open to you. […] one option is that of not performing a meta-analysis. An unwise meta-analysis can lead to highly misleading conclusions.”

The Norwegian evaluations of MST showed that, despite it being a newly implemented program with a control group that did in fact receive an alternative treatment, the program reduced a number of adverse outcomes. These considerations, largely ignored in the Littel review, are important to consider not only because they help researchers and practitioners fine-tune implementation and research procedures, but also because they contribute to the expansion of the knowledge base of program implementation and evaluation.

Some of the shortcomings of the review could have been compensated for by including a number of other family-based treatment methods working with the same target group. The reviewers themselves nod in this direction when they conclude by saying that there is no evidence indicating that any other treatment methods serving youths with serious behavioral problems is more effective than MST.

• Terje Ogden is research director and senior scientist at the Norwegian Center for Child Behavioral Development. He has a degree in education from the University of Oslo, Institute for Educational Research. He is also professor at the Institute of Psychology, University of Oslo. He is currently doing research on empirically validated methods such as Multisystemic Treatment and Parent Management Training in the treatment of conduct problems among children and youth.

• Kristine Amlund Hagen is assistant research director and scientist at Norwegian Center for Child Behavioral Development. She has doctoral degree in child development from Virginia Commonwealth University, USA. Her current research focus is on the development of behavioral problems and social competence in children and youth.

• See also: Norwegian researchers find flaws in ‘gold standard’ program review [1]