We should stop running psychiatric clinical trials that cannot change practice

The case of prazosin for PTSD nightmares and the PACT trial

There are few medications that reflect the messiness of evidence in psychiatry more than prazosin, the old anti-adrenergic blood pressure medication that, as was hounded into me during my training, is synonymous with the treatment of PTSD-related nightmares. My goal here is not to debate the evidence of prazosin, which I’ve written about previously.

Rather, I want to raise a larger question: what are the implications of conducting clinical trials that cannot change practice, no matter their findings?

The large, multisite VA Prazosin and Combat Trauma PTSD (PACT) trial is a case-in-point. Prior to PACT, there were multiple trials of prazosin that showed large effects for reducing nightmares and global PTSD symptoms, but these studies were small, primarily conducted by a single VA research group, and included almost exclusively military populations, many of whom were young, treatment-naïve, and active-duty with direct combat trauma.¹

This led to the uncomfortable situation that, prior to the publication of PACT in the NEJM in 2018, roughly one in four veterans receiving a medication for PTSD in the VA was prescribed prazosin. One in four. This represented about 150,000 veterans per year, and about 40,000 new veterans per year (plus an unknown number of non-veterans, for which there was essentially no research on prazosin). Prior to PACT, this prescribing was based on clinical trials that included about 150 total subjects in the prazosin arms.

Thus, PACT reflected a need for higher quality evidence to justify the extensive prescribing of prazosin. PACT included 304 veterans, roughly as many as every prior trial combined. It was resoundingly negative on its primary outcomes (nightmares, sleep quality, and overall clinical impression) and virtually every secondary outcome. Patients got better with prazosin, but no better than those receiving placebo (placebo actually outperformed prazosin for a chunk of the trial). It was about as negative of a trial as there can be.

From Raskind et al., 2018. CAPS is the standard PTSD rating scale, with the “distressing dreams” item B2 measuring the frequency and severity of nightmares. PSQI is the Pittsburgh Sleep Quality Index, the standard measure of sleep quality with a 0-21 scale.

What interests me is not that PACT was negative, but the fact that this trial has had zero impact on clinical practice, as far as I can tell. This is due to two explanations for the negative results.

Explanation 1: The inclusion criteria were flawed

The most influential explanation was to blame the inclusion criteria. Apparently, the authors selected for the precise patients who don’t respond to prazosin. The patients were too clinically and “psychosocially” stable, as evidenced by low rates of alcohol use (even though the trial excluded those with actual substance use disorders), lower than expected blood pressure, and the exclusion of patients taking trazodone.²

While it’s certainly possible that this could be true, the logic is weak. The authors designed the very inclusion criteria they criticize! They did not view these issues as being problematic a priori. Indeed, the previous positive prazosin trial conducted by this same group used almost identical inclusion criteria.

To highlight the flaws in this reasoning, another negative prazosin study in comorbid PTSD-alcohol use disorder (Petrakis et al., 2016) cited the exact opposite reason for the results. Whereas PACT cited clinical stability as the cause of the negative findings, Petrakis argued that their sample was too clinically unstable to find an effect.

My point is that if PACT were positive, no one would have dismissed the findings because of subjects’ blood pressure, trazodone exclusion, or psychosocial stability. I know this because no one questioned the small positive trial with almost identical inclusion criteria. But, because PACT was negative, we scrutinize the inclusion criteria and say, of course this trial couldn’t work because of X, Y, or Z.

Explanation 2: It works in subgroups, just not the ones studied

The second reaction was to assert that prazosin works in a subset of patients who were not included in the study.

Here again, the evidence is very weak. Yes, a post-hoc analysis of a 67-subject trial showed that prazosin’s effect was blood pressure dependent. But this did not replicate in a post-hoc analysis of the 96-subject alcohol use-PTSD trial above.

The PACT trial clearly could have conducted subgroup analysis to confirm this effect (or run the trial with inclusion criteria for this population to confirm this prospectively). Instead, the authors (and author of the editorial) speculate that this is true, but have never published the data that could validate or falsify this hypothesis. You have to presume that the PACT trial didn’t replicate this either.

Negative results do not prove that the trial is flawed

These arguments have proven so influential that the PACT trial has had no apparent impact on prazosin prescribing.³ Despite being the largest and best trial ever conducted on prazosin, psychiatrists continue to prescribe prazosin exactly as they have before the trial. Indeed, after publication of the trial, the 2023 VA/DoD guidelines strengthened their recommendation for the use of prazosin for nightmares.⁴

This raises the question, why was the trial conducted at all?

Rather than the trial determining whether prazosin works, it seems that we have collectively presumed that prazosin works, so the negative findings are proof that the trial design was flawed and can be disregarded.

If there was already robust evidence in favor of prazosin, it would be fair to ignore a single negative trial. But the existing prazosin data were really weak: small trials from a single research group in very niche populations. It is because the earlier data were so weak that the PACT trial was conducted.

If negative outcomes prove that the trial was flawed, then there is no reason to conduct the trial in the first place. Certainly, there are cases where unforeseen issues arise in a trial that complicate the outcomes, but patients simply meeting the inclusion criteria aren’t unforeseen complications. Any worthwhile clinical trial needs to have jeopardy: we can accept the positive findings only because we would accept the negative findings.⁵

But that’s the other irony with prazosin. Even a positive trial wouldn’t have impacted clinical care. Perhaps a positive trial would have affirmed the use of prazosin, but it was already presumed effective and extensively used in the VA. It was the third most prescribed medication for PTSD and was prescribed more than any class of medications except SSRIs. One in four veterans receiving PTSD medications in a given year were already prescribed it. It’s hard to imagine how a positive trial could have possibly increased its use.

The rebuttal: meta-analyses

Whenever I try to convince other psychiatrist that the PACT trial should change our use of prazosin, I am chastised for ignoring the multiple positive meta-analyses which include the PACT trial. When you look at the sum of evidence rather than just one trial, prazosin clearly works.

There’s much more to say about our reliance on meta-analyses in psychiatry, but for now: this is my very point.

If we are going to blindly defer to meta-analyses to determine the ultimate efficacy of a treatment (as the 2023 VA/DoD Clinical Practice Guidelines for PTSD did), then we should consider what type of negative trial would be needed to influence a meta-analysis before running the trial (I’m talking about large, multi-site trials like PACT that are designed to definitively determine the efficacy of a treatment). Such calculations are not hard to do. If we’re going to defer to the meta-analyses, and if a huge negative trial can’t possibly change their outcomes, then there’s no point in running the trial at all. It literally cannot change practice.⁶

Let me emphasize: the difficulty of changing the outcomes of the prazosin meta-analyses is an artifact of their statistics, and not because of the strength of evidence for prazosin. The prazosin meta-analyses include mostly small trials with large effects. The standard random-effects meta-analytic approach, in essence, overweighs small studies by presuming that each study reflects a distinct effect in that individual trial.⁷ In the Zhang et al. (2020) meta-analysis, for instance, the 304-subejct PACT trial was weighted as 15.6% for the nightmare outcome, while the 13-subject cross-over trial (Taylor et al., 2008) was weighted at 13.6%. The approaches of the other meta-analyses are very similar. No one seriously believes that a 13 person cross-over trial should count as much as the PACT trial, and yet, that’s what the meta-analyses argue.

If you read meta-analyses closely, these absurdities are common. Look at the forest plot below. Petrakis 2016 trial (the 96-subject AUD-PTSD trial I mentioned above) found an astonishingly large 2.7 SMD effect size for prazosin over placebo on nightmares. It’s literally off the chart. In that study, prazosin also apparently massively worsened sleep quality, with a SMD of -1.3. (The actual study was considered negative by the authors, with a tiny clinically irrelevant effect on nightmares for prazosin. All of this is all an artifact of unusually small standard deviations.) I’ll stop myself from telling you about the other studies in this meta-analysis, but spoiler: they’re not great!

Zhang et al., 2020 random-effects meta-analysis for prazosin vs placebo for nightmares, sleep quality, and PTSD symptoms. Outcomes are reported as standardized mean differences, basically the number of standard deviations that separate prazosin and placebo. For reference, antidepressants outperform placebo by ~0.3 SMDs for depression.

A zombie trial

Dr. John Carlisle termed trials with falsified data or other fatal flaws “zombie trials:” they appeared to be normal studies, but were dead inside.

The PACT trial represents a different type of zombie trial: a real trial, but unable to influence practice. It exists as a curiosity in the academic literature, but has no clinical relevance. In my experience, most practicing VA psychiatrists and sleep medicine doctors are unaware that the largest trial of prazosin, by far, was negative.

We will probably never get another large trial of prazosin for nightmares. A decent-sized German study of clonidine, doxazosin (a longer-acting alpha-1 antagonist), and placebo was recently stopped due to poor recruitment. It’s an old generic medication that costs pennies. So the best evidence we will ever have is the PACT trial, and clinical practice has not changed at all.

This is disappointing because PACT has value for clinicians who accept its findings. It suggests that PTSD symptoms, sleep quality, and nightmares are responsive to non-specific non-pharmacological effects (as shown by Sugarman et al. and countless other PTSD trials, like losartan for PTSD and the accompanying editorial). Thus, in clinical practice, the medication seems to work because patients get better with an explanatory model, positive expectancy, and clinical contact. This is an important and compelling finding that should shape our understanding of PTSD treatment.

This is not unique to prazosin

I focus on the PACT trial because I know the prazosin data well. But it’s hardly alone. There are numerous major negative trials in veterans that have had no apparent impact on the clinical care of veterans (e.g. risperidone LAI versus oral antipsychotics for schizophrenia, naltrexone for alcohol use disorder, and sertraline for PTSD, to name a few). Outside of veterans, there are numerous negative trials for specific populations which don’t impact practice at all, like antidepressants for depression in alcohol use disorder, or antidepressants for depression in heart failure or chronic kidney disease.

My favorite example is the SADHART-CHF trial, which showed no benefit for sertraline over placebo for depression in heart failure, and then recommended sertraline anyway. What, then, is the value of running these studies? A positive trial would affirm the existing, widespread prescribing pattern, and a negative trial doesn’t change practice at all.

From the SADHART-CHF trial (O’Connor et al., 2010), which recommended sertraline anyway.

In a moral sense, it should worry us that people are volunteering for clinical trials and subjecting themselves to blinded experimental treatment for no clear reason. It also means that scarce resources are being allocated to clinical trials that could be better used elsewhere.

More broadly, I see a worrying trend in psychiatry where studies influence practice if they are positive, but are largely disregarded if they are negative. The Henssler 2022 meta-analysis on combination antidepressants that Nils wrote about was widely publicized for its support of SSRI-mirtazapine combinations, but the negative finding for bupropion augmentation has mostly been ignored, even though this reflects a practice-changing finding. There are high-quality meta-analyses that find that common antidepressant treatment steps are no better than the control, which have had almost zero impact on clinical care. Negative studies seem to receive detailed critical appraisal to explain away negative findings, while even the smallest trials with major methodological flaws can influence practice.

It’s uncomfortable to have a promising treatment fail in a large clinical trial, particularly when there aren’t obvious alternative treatment options, as with PTSD nightmares. I see veterans with nightmares routinely, and it’s demoralizing to not have better ways to help them. But if we’re intent on finding treatments that are better than placebo, we need to acknowledge negative evidence and insist on prescribing treatments that truly work. Otherwise, it seems that we’re just giving our patients active placebos for treatments that are responsive to expectancy, as the PACT trial showed.

This is what it means to practice medicine and to rely on evidence. And what it means to be honest with our patients and ourselves.

1. That is, not remotely generalizable to the average VA patient, much less civilians.

2. The exclusion for social instability which was discussed as a primary reason for the negative trial is not even specifically listed as causing any subjects to be excluded in the supplement. Among the 413 veterans screened, 103 were deemed ineligible but no patients are specifically excluded for social instability (31 “per clinical judgement” and 5 for “other”). Could this really cause the entire 304-subject trial to be negative? Not to mention, if prazosin only works in patients with elevated blood pressure or heart rate, then just make that an inclusion/exclusion criterion.

3. Unfortunately, the best study evaluating trends in PTSD medication prescribing in the VA ended in 2018, when PACT was published.

4. This simplifies things slightly in the name of space. The 2017 guidelines made no recommendation, but used clinicaltrials.gov access to PACT (I kid you not) to see that the trial was negative, and this influenced their guidelines (PACT oddly went unpublished for several years after completion). The 2023 guidelines were influenced by meta-analyses, as discussed below.

5. This is the same essential problem with reporting bias, p-hacking, hypothesizing after results are known (HARKing) and other bad research practices that have led to the replication crisis in many social sciences. You don’t get an endless number of chances to slice and dice data to find a positive result, after your primary outcomes were negative. You have to declare what you believe and are testing for upfront, and then stick to it. This is why we now preregister studies, publish protocols, and so forth.

6. This was my frustration with the Henssler meta-analysis about combination antidepressants, which Nils wrote about. The small study effects are so large that the highest-quality, big negative trials don’t even matter.

7. In essence, random-effects meta-analyses are based on the assumption that there is no “true” treatment effect. The “true” effect of the medication would depend on the specific population and treatment parameters in each study. Therefore, even small studies should be weighted heavily because they are capturing a unique effect in that specific population. By contrast, fixed-effects meta-analyses assume that there is a single “true” effect and basically weight a study based on sample size/variance. There is much more nuance about which approach is applicable based on the number and size of individual trials. Almost every psychiatric meta-analysis uses a random-effects approach, which is usually considered the more conservative approach.