The null hypothesis wins again. I am sorry to inform you Dr. Kennedy, but you are clearly cursed to live a long life of scientific thinking while surrounded by unscientific mumbo jumbo.
I was surprised to learn that psych NP exams make prazosin the right answer to test questions about PTSD. Never mind that prazosin is not FDA approved, the evidence is not there beyond placebo, and the mechanism of action is iffy.
So another placebo for PTSD is here to stay. I put prazosin right next to eye movements.
However, the trial did change at least one clinical practice. I can report to you that I no longer introduce prazosin to patients.
I'd go completely the opposite direction of meta-analysis and argue that moderate to large signals shouldn't require a large sample and may not even require statistical analysis depending on just how strong of a signal is being discussed. If a small trial gets a particularly strong signal that vanishes in a larger trial, it suggests confounders to me. Of course, it depends very much on how strong that original signal was, but it's important to distinguish, in my view, between moderate to marginal effects that require greater statistical power and strong effects that may not require much statistical power at all.
I'm talking here about things that pass the interocular trauma test, i.e. results that hit you right between the eyes. It's alarming to me that most psychiatrists in this situation would reach for the meta-analysis.
Thanks for the comment. I'm working on a piece about our reliance on psychiatric meta-analyses and a piece on small studies with large effects, so I'll save some detailed thoughts for those.
There's been a fair bit written about this, e.g. needing RCTs to prove that parachutes work or RCTs for penicillin in the Fleming era. It's also the argument of David Healy who points out, in the opposite direction of you, that the need for large samples highlights how small most effects are.
The reality is that the effects of most treatment vs placebo in psychiatry are small. I'm curious to hear which ones you would view as passing the intraocular trauma test. I can think of a few, e.g. Ativan for catatonia, but not a huge number. There are lots of examples of strongly held beliefs in psychiatry that were proven wrong in RCTs, like the need to titrate antipsychotic doses to the neuroleptic threshold. The original clozapine RCT involved mean Haldol doses of 61mg.
My concern is that it quickly becomes non-falsifiable to basically argue that once a small trial detects a large effect, then any replication that fails to detect that effect is confounded. If effects are large, then they should replicate.
Your idea that a worthwhile trial must contain real jeopardy may need to extend beyond the investigators.
We preregister hypotheses, outcomes, and analyses so the scientific question can't be rewritten once the result is known. But the field rarely commits, in advance, to what any given result would oblige it to do. What null finding would trigger a guideline review? What magnitude would actually reduce prescribing? What result would mean another trial isn't worth asking patients to enter?
Without that commitment, a negative trial can always be absorbed after the fact — by a fresh subgroup theory, a revised reading of the inclusion criteria, or a meta-analysis whose decision rule was never fixed beforehand. The interpretation gets written after the data, which is exactly what preregistration was meant to prevent everywhere else.
So perhaps a trial needs not only a statistical analysis plan but an interpretive consequence plan: a statement, written before enrollment, of what each plausible result would commit the field to reconsider.
Yep, all the above, and. . . below. I would suggest some serious clinical work on the upside and downside of clonidine and guanfacine for modulating the sensory amplification that an "overactive" locus coeruleus can cause. Just a thought. As for EMDR, it works well if you have decent enough working memory capacity. Without baseline optimal WM capacity, it is very difficult to "overwrite" old memory traces of trauma/threat. It is likely that screening for low working memory (ADHD) and appropriately enhancing WM with a correct dopamine enhancer medication would have an effect on the efficacy of EMDR for trauma memory modification.
627 veterans randomized. 3 arm trial. I'll just quote their conclusion:
"In this large, multisite study, in which we used the same alcoholism outcome measures that were employed in earlier single-site studies, we did not detect an effect of naltrexone. Relative to placebo, naltrexone did not prevent or delay relapse to heavy drinking, reduce the number of drinking days, or decrease the amount of alcohol consumed during episodes of drinking. Major outcomes were not influenced by the duration of naltrexone administration, the degree of compliance with study medication, participation in counseling sessions, or attendance at Alcoholics Anonymous meetings. Our data do not support the treatment of alcohol dependence with naltrexone combined with a psychosocial treatment program in men with chronic, severe alcohol dependence."
The null hypothesis wins again. I am sorry to inform you Dr. Kennedy, but you are clearly cursed to live a long life of scientific thinking while surrounded by unscientific mumbo jumbo.
I was surprised to learn that psych NP exams make prazosin the right answer to test questions about PTSD. Never mind that prazosin is not FDA approved, the evidence is not there beyond placebo, and the mechanism of action is iffy.
So another placebo for PTSD is here to stay. I put prazosin right next to eye movements.
However, the trial did change at least one clinical practice. I can report to you that I no longer introduce prazosin to patients.
There are dozens of us, dozens!
(Since I can't figure out how to post gifs: https://giphy.com/gifs/kSlJtVrqxDYKk)
I'd go completely the opposite direction of meta-analysis and argue that moderate to large signals shouldn't require a large sample and may not even require statistical analysis depending on just how strong of a signal is being discussed. If a small trial gets a particularly strong signal that vanishes in a larger trial, it suggests confounders to me. Of course, it depends very much on how strong that original signal was, but it's important to distinguish, in my view, between moderate to marginal effects that require greater statistical power and strong effects that may not require much statistical power at all.
I'm talking here about things that pass the interocular trauma test, i.e. results that hit you right between the eyes. It's alarming to me that most psychiatrists in this situation would reach for the meta-analysis.
Thanks for the comment. I'm working on a piece about our reliance on psychiatric meta-analyses and a piece on small studies with large effects, so I'll save some detailed thoughts for those.
There's been a fair bit written about this, e.g. needing RCTs to prove that parachutes work or RCTs for penicillin in the Fleming era. It's also the argument of David Healy who points out, in the opposite direction of you, that the need for large samples highlights how small most effects are.
The reality is that the effects of most treatment vs placebo in psychiatry are small. I'm curious to hear which ones you would view as passing the intraocular trauma test. I can think of a few, e.g. Ativan for catatonia, but not a huge number. There are lots of examples of strongly held beliefs in psychiatry that were proven wrong in RCTs, like the need to titrate antipsychotic doses to the neuroleptic threshold. The original clozapine RCT involved mean Haldol doses of 61mg.
My concern is that it quickly becomes non-falsifiable to basically argue that once a small trial detects a large effect, then any replication that fails to detect that effect is confounded. If effects are large, then they should replicate.
Your idea that a worthwhile trial must contain real jeopardy may need to extend beyond the investigators.
We preregister hypotheses, outcomes, and analyses so the scientific question can't be rewritten once the result is known. But the field rarely commits, in advance, to what any given result would oblige it to do. What null finding would trigger a guideline review? What magnitude would actually reduce prescribing? What result would mean another trial isn't worth asking patients to enter?
Without that commitment, a negative trial can always be absorbed after the fact — by a fresh subgroup theory, a revised reading of the inclusion criteria, or a meta-analysis whose decision rule was never fixed beforehand. The interpretation gets written after the data, which is exactly what preregistration was meant to prevent everywhere else.
So perhaps a trial needs not only a statistical analysis plan but an interpretive consequence plan: a statement, written before enrollment, of what each plausible result would commit the field to reconsider.
Yep, all the above, and. . . below. I would suggest some serious clinical work on the upside and downside of clonidine and guanfacine for modulating the sensory amplification that an "overactive" locus coeruleus can cause. Just a thought. As for EMDR, it works well if you have decent enough working memory capacity. Without baseline optimal WM capacity, it is very difficult to "overwrite" old memory traces of trauma/threat. It is likely that screening for low working memory (ADHD) and appropriately enhancing WM with a correct dopamine enhancer medication would have an effect on the efficacy of EMDR for trauma memory modification.
Great article! What’s the evidence for the negative effect of naltrexone AUD?
The study I referenced was Krystal et al., 2001 https://www.nejm.org/doi/full/10.1056/NEJMoa011127
627 veterans randomized. 3 arm trial. I'll just quote their conclusion:
"In this large, multisite study, in which we used the same alcoholism outcome measures that were employed in earlier single-site studies, we did not detect an effect of naltrexone. Relative to placebo, naltrexone did not prevent or delay relapse to heavy drinking, reduce the number of drinking days, or decrease the amount of alcohol consumed during episodes of drinking. Major outcomes were not influenced by the duration of naltrexone administration, the degree of compliance with study medication, participation in counseling sessions, or attendance at Alcoholics Anonymous meetings. Our data do not support the treatment of alcohol dependence with naltrexone combined with a psychosocial treatment program in men with chronic, severe alcohol dependence."
This is clearly an AI written comment. This is your first and only warning to not do so again.