Norwegian researchers find flaws in 'gold standard' program review

Establishing whether an intervention is effective can be doubly difficult if doubts are cast on the quality of the evaluation work commissioned to settle the matter – and one argument seems to only to provoke another.

Studies of the reliability of Multisystemic Therapy (MST), soon to be trialled in England (see $14m for UK pilots of Multisystemic Therapy Services Inc. [1]) are a case in point. Randomized controlled trials of MST have been generally encouraging – hence the UK government’s confidence in piloting the program as a well-grounded response to the needs of young people with complex clinical, social, and educational problems.

But in 2005 a systematic review of eight initiatives for the Cochrane Collaboration by Julia Littell and colleagues, which might have been expected to clinch the argument, failed to find any conclusive evidence that MST was any more effective than any other intervention.

The disappointment and irritation of Multisystemic Therapy's US designers was predictable, but doubts about the value of Littell’s findings have also been raised in Norway – where MST has been widely trialled – by Terje Ogden and Kristine Amlund Hagen of the Norwegian Center for Child Behavioral Development.

They argue that the foundation on which the Cochrane review was based was insufficient and its lukewarm conclusions consequently premature. Although eight MST evaluations were included, they say that the analysis was confined in most cases to only three.

"We believe that these numbers seriously limit the ability to draw any general conclusions about MST. Most meta-analyses observe at least 50-200 studies in order to have sufficient power to make general statements about effects."

Littell and her team themselves acknowledged this weakness in 2005 stating: "Since statistical power is low, we can not conclude that MST is not more effective than other services."

But Ogden and Hagen regard this admission as going to the heart of the problem, arguing that it illuminates the shortcomings of the review itself rather than any limitations of Multisystemic Therapy.

The Norwegian also question the inclusion in the review of an unpublished Canadian study which weighs heavily because of its large sample number but which failed to undergo the quality assurance system that a peer review process provides.

Furthermore, they say they detect a logical flaw in Littell's team's criticism. "When they disapprove of the methods, data, and design of the investigations included in their meta-analysis, they simultaneously undermine their own study: A meta-analysis can never be better than the investigations on which it is based."

They conclude by suggesting that the Cochrane Collaboration itself is on their side of the argument, citing its own advice to reviewers: "If you identify and suspect that important diversity or heterogeneity is present in your review, there are several options open to you. … one option is that of not performing a meta-analysis. An unwise meta-analysis can lead to highly misleading conclusions."

•For more on Terje Ogden and Kristine Amlund Hagen's response to the Littell team's report, see Can a systematic review be any better than the work it’s based upon? [2]