LifeSkills Training, which claims to reduce the use of marijuana, tobacco, and alcohol use among teens, is one of the most well known and most popular drug prevention programs. LST has been delivered to an estimated three million middle school students in the USA alone. It has been used in over 30 countries, including the UK.

And it’s not just popular; it’s proven – for example, it is listed on many “effective program” databases, including the prestigious Blueprints Model Program list, because of the wealth of evidence supporting its effectiveness.

So we can be confident that it works as advertised. Right?

Wrong, says new analysis from Texas A&M University. In fact, taking a closer look at the trials that have “proven” the value of LST, the majority show LST to have no effect at all on marijuana use.

This is the conclusion reached by professor of epidemiology and biostatistics Dennis Gorman. Gorman examined eight evaluations of LST – five by the program developer and three by other research teams. He finds that most of the results show no effect. What’s more, even the small number of findings that show LST having an effect are dubious: some of the positive findings might be “dependent on the form of data analysis that was conducted.”

For example, in the studies conducted by the program developer, there were a total of 41 comparisons between those receiving LST and a control group. Only seven of the 41 comparisons, Gorman found, showed statistically significant reductions in marijuana use. Even some of these seven, he says, are marginal. And the remaining 34 comparisons showed no effect either way.

Do drug prevention agencies ignore “no effect” results?
Looking at these results, even the most optimistic advocate of LST would question the intervention’s effectiveness. This, of course, begs the question: why is LST still seen by drug prevention agencies as effective at reducing marijuana use?

Gorman points the finger at those who, as he puts it, “pay little attention to the preponderance of evidence pertaining to the absence of an effect on marijuana use.” He notes, for example, that SAMHSA’s National Registry of Evidence-based Programs and Practices highlights the effect of LST on marijuana use based on just two studies.

Other officials also need to be more mindful in their reporting, Gorman suggests. He quotes a report by the US Department of Education that says that “almost every study [of LST] showed significant results that favored the treatment group.” This statement is factually true – “almost every study” produced a few statistically significant results on one measure or another – but it glosses over how consistent the findings were, or what the measures were. And in the case of marijuana use, supportive results are the exception – not the rule.

And Gorman suggests a further reason for LST’s popularity despite a relative lack of evidence: the age of the studies. It’s possible that adolescent drug cultures have evolved over the past two or three decades. Most of the statistically significant differences were reported in publications 20 or more years old. Perhaps LST was more effective in the past.

A stricter analysis
Gorman’s main focus, however, is on the way in which previous analysts’ methods may have skewed the results in favor of LST. He aims to create a stricter and clearer analysis of the previous trials. Unlike some previous studies, Gorman analyzed the full group of participants, not a subgroup.

In the case of the five trials conducted by Gilbert J. Botvin, who developed LST, control groups were compared with a subset of the intervention group. Only those who received 60% or more of the program dosage were reported. This meant that around one-third of those randomly assigned to the intervention group were excluded from analysis – resulting in self-selection bias and undermining the purpose of randomization. This approach to analysis, according to Gorman, unfairly favored LST.

Several studies of LST, conducted independently of the program developer, also found a significant impact of LST. Here, Gorman claims, a more detailed examination of the analysis turns up similar methodological issues. One significant result was found during a final follow-up assessment, at which point only 38% of participants remanded in the study – severely limiting the evaluation’s validity. Another decided only to report findings only on a female subgroup, and not on the male participants, again likely biasing the sample in favor of LST.

Warning for policy and practice
The author finishes the article with implications for research policy and practice. Perhaps the most urgent implications are the ones for school-based drug prevention practitioners: “If one of their primary goals is to reduce marijuana use, then it would behoove them to look to interventions other than the LST program.”

For drug agencies, those who maintain databases of effective interventions, and the journals that report on such evaluations, Gorman points out the need to develop more consistent standards and procedures of data analysis and reporting, so that differences in program efficacy aren’t masked by the way in which data are analyzed.

Gorman’s article acts as a warning to scientists and reviewers, too. Even a significant result unearthed by randomized control trials – the “gold standard” of evaluation – is not enough to make a program “proven.”

References
Gorman, D. “Does the Life Skills Training program reduce use of marijuana?”. Addiction Research and Theory (2011), 1-12. DOI: 10.3109/16066359.2011.557164

Links:
www.lifeskillstraining.com