Wednesday, December 26, 2012

How to Discredit Research on MDMA's Benefits: First, Misread the Abstract

How to Discredit Research on MDMA's Benefits: First, Misread the Abstract:
Sepkowitz, an internist and infectious disease specialist at
Memorial Sloan-Kettering Cancer Center in New York City, thinks
people have too readily accepted the idea that MDMA can be useful
as a psychotherapeutic catalyst for soldiers and others diagnosed
with posttraumatic stress disorder. In particular, he
in a Daily Beast essay, the press and
public should be more skeptical of
sponsored by the Multidisciplinary Association for
Psychedelic Studies (MAPS) in which subjects who took MDMA were
much more likely to improve (as measured by scores on the Clinician
Administered PTSD Scale) than subjects who took a placebo.
That result, Sepkowitz claims, was "suggestive but not
statistically significant," because "the likelihood that the result
was due to the drug was 83%," as opposed to 95 percent, the usual
standard for statistical significance.
But as MAPS Executive Director Rick Doblin points out in the comments below
Sepkowitz's article, the doctor, a professor of medicine at Weill
Cornell Medical College, seems to have made an embarrassing
error: Eighty-three percent is the share of subjects in the
treatment group who improved, compared to 25 percent in the control
group. The difference was indeed statistically significant, with a
1.3 percent probability that it would occur purely by chance; to
put it another way, the probability that the difference reflects a
genuine drug effect (assuming the study was in other respects
properly designed) was 98.7 percent, not the 83 percent that
Sepkowitz claims. Doblin comments:
Kent seems to have been so motivated to critique our initial
study that he misread our abstract and reported an incorrect claim
that our statistical analysis was not significant....If Kent had
taken more than a few minutes to read the abstract and thought
about what it actually reported, or if he had decided to read the
actual paper, or even part of it, he would have learned that we
reported that the difference between a response rate of 83% in the
MDMA group and 25% in the placebo group (whose subjects received
our extensive psychotherapy but with an inactive placebo) was
significant ([p =] 0.013).
What about Sepkowitz's more general claim, that people are
making too much of a suggestive but small and inconclusive study,
partly because they are attracted by "the hipness factor" and a
story line in which "Ecstasy leaves the gutter to save the
day"? First of all, MDMA was arbitrarily consigned to "the
gutter" by the Drug Enforcement Administration, which imposed an
"emergency" ban on the substance in 1985 after people began using
it for fun (shudder) at clubs and dance parties. But before MDMA
emerged in that recreational context, it was a hit among
psychotherapists who found that it enhanced empathy and candor (and
who opposed its prohibition for that reason). While their
testimonials are not the sort of evidence that gets pharmaceuticals
approved by the Food and Drug Administration (FDA), it's not as if
the idea of using MDMA in psychotherapy came out of the blue.
Now that MAPS is jumping through the hoops
necessary to win FDA approval, Sepkowitz warns that more evidence
is needed. No kidding. The title of the first report on this
study, which was published two years ago in the Journal of
, was "The Safety and Efficacy of
3,4-Methylenedioxymethamphetamine Assisted Psychotherapy in
Subjects With Chronic, Treatment-Resistant Posttraumatic Stress
Disorder: The First Randomized Controlled Pilot Study." The authors
(who included Doblin as well as the lead investigator, South
Carolina psychiatrist Michael Mithoefer) concluded that
"MDMA-assisted psychotherapy can be administered to posttraumatic
stress disorder patients without evidence of harm, and it may be
useful in patients refractory to other treatments." They added,
"The promising results of this initial pilot study suggest
that further research is warranted to confirm our findings,
distinguish and refine the essential elements of this
approach, enhance the methodology, and elucidate the
mechanisms involved."
A follow-up
published in the same journal last month (which
generated the press coverage that apparently offended Sepkowitz)
reported that "the majority of these subjects with previously
severe PTSD who were unresponsive to existing treatments had
symptomatic relief provided by MDMA-assisted psychotherapy
that persisted over time, with no subjects reporting harm from
participation in the study." The authors added, "Should further
research validate our initial findings, we predict that
MDMA-assisted psychotherapy will become an important treatment
option for this very challenging clinical and public health
problem." Or as Sepkowitz puts it, "If confirmed by larger trials,
this finding would be welcomed."
Contrary to Sepkowitz's implication, the researchers did not
suggest this was anything more than a pilot study with a small
number of subjects (a total of 20 initially). In fact, they
emphasized this was the first study of its kind to test
the psychotherapeutic potential of this prohibited substance (a
precedent that helps explain why it attracted
press attention
), and they emphasized the need for further
research to replicate their findings (as Doblin does again in his
comment at The Daily Beast). Perhaps some news reports on
this research exaggerated its import, but Sepkowitz does not cite a
single specific example.
Sepkowitz correctly warns that bias can lead to flawed research
that confirms an investigator's preconceived notions, and he
himself illustrates that danger with his inaccurate description of
the initial MDMA study. Certainly it's true that Doblin and
Mithoefer want to see certain kinds of results, but that is hardly
an unusual situation among researchers. As Doblin observes,
scientists have developed safeguards aimed at minimizing the impact
of bias, including randomization, double-blind procedures,
transparent descriptions of their methods, and pre-publication peer
review. Post-publication criticism is another important safeguard
against bias, but it should be done more carefully and thoughtfully
than Sepkowitz manages here.
Speaking of bias, my own view is that MDMA, like other currently
illegal drugs, should be available to any adult who wants it for
any reason, including fun as well as self-improvement, regardless
of what research indicates about its benefits and whether or not
the user has managed to obtain a psychiatric diagnosis. But I
admire the efforts of Doblin and his colleagues to win a bit more
pharmacological freedom by working within the system, a quest that
seemed quixotic when MAPS was founded 26 years ago, right after the
DEA banned MDMA. Despite my sympathy for their cause, I agree with
Sepkowitz that their research should be judged by its scientific
merits and should not get a pass simply because it may help loosen
restrictions on irrationally proscribed substances. But by the same
token, researchers should not have to jump a higher hurdle simply
because their work casts doubt on our government's irrational bias
against certain psychoactive chemicals.