The Sober Truth (7 page)

Is it possible that the kind of people who stay in 12-step programs are
already
more likely to improve? Would they be equally likely to do so in any treatment, or even no treatment at all? At heart, the dilemma facing AA research is whether people stay in AA because they’re the type of people who will stick with a program no matter what it is and who would have stuck with it even if it were of no help to them at all.

In 2005, husband-and-wife team Rudolph and Bernice Moos of Stanford University published the first of two papers that would become some of the most-cited data in support of Alcoholics Anonymous.
¹⁸
Because these articles have become major sources of faith in the effectiveness of AA, they deserve an especially careful review.

The authors conducted a longitudinal, observational study of 362 previously untreated people who chose to enter either AA, professional treatment, or a combination of both. Notably, the authors never defined what was meant by “professional treatment,” or the level of training or competence of the professionals performing the treatment, a point they conceded in the 2006 paper, “Participation in Treatment and Alcoholics Anonymous”: “[An] issue involves the lack of data on the content of treatment, which might have enabled us to examine whether aspects of psychological and social functioning changed less because they were not addressed adequately in treatment.”

In truth, the word
treatment
could mean almost anything in this context, including the very real possibility that it was 12-step-based as well, or was “motivational enhancement therapy,” which is a brief encouragement-based approach that does not resemble serious psychotherapy. The paper’s definition of “long-term treatment” is also mistaken. The researchers defined this as anything more than six months, while most well-trained and experienced professionals in psychology would consider that a short-term treatment.

Surveys were sent out at various checkpoints: one, three, eight, and sixteen years. In their first paper, the researchers concluded:

It was, in other words, a mixed bag. Visit a therapist and AA together, the data suggests, and you are likely to do better than you would with therapy alone. But visit a therapist for one year and
then
try AA, and you won’t do any better than if you had just stayed in therapy.

Notably, the researchers went on to publish a far more strongly worded follow-up in 2006, drawing from the same data. This paper begins by demonstrating some similarities in compliance with treatment between the AA attendees and “treatment” group:

Unsurprisingly, they found that the people who stuck with
either
treatment—AA or professional treatment—did significantly better than those who did not. These were the compliers. The authors continue:

In other words, again unsurprisingly, they found that the people who stuck with either approach—AA or professional treatment—did significantly better than those who did not. Yet the last sentence suggests that people continued to improve over time with AA, whereas they failed to continue improving with treatment. (The authors measured improvement via self-reports in answer to questions such as “Have you been sad the past month?” or “Have you participated in social activities?”) What their conclusion doesn’t address, however, is the possibility that the people in treatment were
already
doing better than the AA group, and that they therefore had less room to improve over those last eight years. We do not know, nor do the researchers say, which interpretation is right.

More problematic is that the study elided some potentially telling fluctuations in the data. People who stayed in AA for fewer than six months had w
orse
outcomes than people who never entered AA at all. This finding seems to mirror the Brandsma data: AA attendees seem to get worse before they get better. One theory is that the finding is nothing but
noise
—the standard statistical turbulence that can foul any short-term study. But if the data are real and repeatable, then they suggest something the Moos researchers perhaps did not consider: that AA might do more harm than good for the people who
choose
to attend but do not
buy into
the program.

The Moos study also employs some objectionable statistical methods. In one critical omission, its conclusions ignore all the people who died and the large number of people who dropped out of the study altogether, despite conceding that these were the people who statistically consumed the most alcohol. As early as year eight, the number of subjects who were left in AA had already shrunk by nearly 40 percent (from 269 to 166), yet these people are erased from the conclusions as if they had never existed at all. Add up all the people who died and the dropouts, and the results for AA become far grimmer than the authors suggest.

The stated size of this survey is also misleading. Although the researchers began with 628 people, the total number of people who remained through the sixteen-year follow-up
and
also stayed in AA for longer than six months—that is, the group on which the authors’ major findings are based—was just 107, or 17 percent of the original sample. And of the remaining 107, the researchers never revealed the actual number of people who improved, or even stayed sober. They told us only which group “had better outcomes.”

Next, there is the question of validity of the results. As I have mentioned, self-reporting is a tricky methodology, prone to the illusions of self-deception and imperfect memory. In most observational research, surveys are the standard currency—without surveys, there can be no data. Yet there are ways to mitigate the information people report about themselves, notably independent testing. The Moos study did not attempt to independently verify any of the surveys it was based on. (The Fiorentine group, by contrast, supplemented their surveys with urine tests.)

Finally, the punctuated nature of the study addressed only the six-month windows prior to each of the four check-ins. This meant that of the sixteen years covered by the study, the researchers’ surveys gathered information on only two of them. No questions were ever asked about the stretches of time in between follow-ups; 88 percent of the time was never studied. As the authors acknowledged in the 2006 paper, “Another limitation is that we obtained information only on 6-month windows of alcohol-related outcomes at each follow-up, and thus cannot trace the complete drinking status of respondents over the 16-year interval.”

Ultimately none of these issues should be great enough to disqualify the Moos study on its own. But together they should give us pause. The study had no controls, so subjects were free to join and leave treatment as they wished. And for every slice of subjects that got better, the study omitted many about which we are never told a thing. Possibly as a consequence of these limitations, the authors of the study readily acknowledged that they, too, struggled with the question of cause and effect:

Why do large observational studies such as that of Fiorentine and Moos seem to suggest that AA is effective, while smaller controlled studies like those of the Brandsma, Walsh, and others included in the
Cochrane Review
do not? The likely explanation is simple: people stay in AA if they’re getting better and leave if they aren’t. This is understandable. If you are able to stop drinking, then continuing to attend AA is a comfortable and affirming choice. If you struggle with drinking and can’t make use of the AA approach, then you are less likely to keep attending. Over the long term, the people who remain in AA are, by definition, the success stories. But they represent a very small slice of the people who start there; as we will see shortly, the dropout rate from AA is extremely high.

These facts—that AA works for the diehards and fails for the dropouts—are perennially misunderstood by the press and even by some researchers. Proponents of the program proudly point to the fact that people who stay in AA tend to be sober, ignoring the tautological nature of this claim. Reviewing this logic, Harvard biostatistics professor Richard Gelber said, “The main problem is the self-fulfilling prophesy: the longer people stick with AA the better they are, hence AA must be working. It is like saying the longer you live, the older you will be when you die.” As we will soon see, this fundamental error in logic undergirds nearly every claim of AA’s efficacy.

Despite the known limitations with the Moos data set, a number of researchers have used it to publish pro-AA papers of their own. For instance, in 2008, J. McKellar (writing as lead author, with Ilgen, Moos, and Moos as coauthors) concluded that “clinicians should focus on keeping patients engaged in AA.”
²²
This recommendation is even more dogmatic than Moos and Moos suggested in their original paper. In fact, this paper itself notes that pressuring people to attend AA is usually unhelpful: “a significant number of substance abuse patients never attend self-help groups after discharge,” that is, when no longer mandated to attend.

In 2011, again, using the original Moos data, Stanford’s Christine Timko published as lead author on another paper with Moos and Moos, drawing a similar conclusion:

Another observational AA study was conducted by John-Kåre Vederhus in 2006, once again without randomization or any interventions. This one looked at a small group—just 114 patients—with drug and alcohol problems, and found the same broad correlation as Moos and Moos: Intention-to-treat-analysis showed that 38 percent still participated in self-help programs two years after treatment. Among the regular participants, 81 percent had been abstinent over the previous 6 months, compared with only 26 percent of the non-participants.
²⁴

Once again, after two years, over 60 percent of the people remaining in the study had dropped out of AA. The people who stayed were admirably sober, on the order of 81 percent. But the total number of people who were sober and still attending AA was only 31 percent of the whole group. Despite this figure, and despite the fact that the study involved only people who had self-selected into 12-step programs and that nearly 35 percent of subjects had dropped out entirely, biasing the results toward more positive outcomes, the authors state, “We conclude that the probability of a positive effect is sufficient to recommend participation in self-help groups as a supplement to drug addiction treatment.” This study was repeated in our popular press as proof positive that AA was a success.

In 2012, yet another longitudinal and observational study, conducted in Sacramento, California, by Jane Witbrodt and colleagues, found essentially the same results as the prior studies, namely, that the people who are still in AA at the end of many years tend to be admirably sober and well.
²⁵
Once again, however, there were familiar issues: 25 percent of the study subjects had dropped out by year 9, and of those remaining, only 25 percent were “high” attenders of AA, which was the group with the best outcomes. (Even within this high-attending group, 22 percent were still drinking.) Like its cousins, this study relied on self-reporting and, like its cousins, acknowledged a major caveat: “We suspect that the higher abstinence in our ‘high’ class may be due in part to this being a more stably insured or employed population.” The authors also acknowledged that they “lacked baseline measures for prior 12-step involvement and treatment episodes. Undoubtedly, these prior exposures may have influenced subsequent attendance for some study participants.” It appears from this acknowledgment that the authors were aware that there was a pro-AA bias in the selection of their sample.

In fact, every one of the subjects in this paper had already been through the eight-week Chemical Dependency Recovery Program at the Kaiser Permanente facility in Sacramento, which is a 12-step-based program. In other words, all of the subjects of this study had already been exposed to the AA philosophy and actively encouraged to attend before they were followed up to determine if AA treatment would be helpful. It would be hard to imagine a clearer example of selection bias. If the authors had titled their article “Abstinence among People Intensively Exposed to AA Doctrine Who Then Chose to Continue with AA,” they would have been on more solid ground.