In 1948, the behavioralist B.F. Skinner published “‘Superstition’ in the Pigeon.”1 Skinner devised a contraption where he took starving pigeons, put them in a cage, and attached a food hopper that swung into the cage at timed intervals. Here’s what happened next:
One bird was conditioned to turn counter-clockwise about the cage, making two or three turns between reinforcements. Another repeatedly thrust its head into one of the upper corners of the cage. A third developed a ‘tossing’ response, as if placing its head beneath an invisible bar and lifting it repeatedly. Two birds developed a pendulum motion of the head and body, in which the head was extended forward and swung from right to left with a sharp movement followed by a somewhat slower return.
Skinner described the process of conditioning he observed:
The bird happens to be executing some response as the hopper appears; as a result it tends to repeat this response. If the interval before the next presentation is not so great that extinction takes place, a second ‘contingency’ is probable. This strengthens the response still further and subsequent reinforcement becomes more probable.
This, dear readers, is how many psychiatrists seem to practice.2
When I write about treatments that are no better than placebo in large clinical trials, the standard rejoinder is “Well, I’ve had some patients get better with it” or “I’ve had good luck with it” or “You just have to get up to a really high dose.” It reminds me of the film Sideways where Miles (Paul Giamatti) gives a detailed critique of a wine only for his friend Jack to respond, “I don’t know, tastes pretty good to me.”
The underlying belief is that psychiatrists are better at detecting treatment effects than clinical trials or, alternatively, that they can recognize distinct populations in which a medication is effective, even if the medication didn’t work in the clinical trial as a whole.
And I can understand this: seeing a patient do better after a medication change is powerful and affirming. The patient was doing poorly, you made a change, and now she’s doing better. And the patient herself is reporting that the medication change caused the improvement. In psychiatry, there are few more things more ego-inflating than figuring out the right medication regimen for a patient.
But it doesn’t prove that this is right. Being a scientific discipline, we should test this hypothesis: can psychiatrists detect specific pharmacological effects in an individual patient, beyond that of another treatment? Let me offer some examples.3
Example 1: Esketamine augmentation for depression
Let’s start with one of my favorite figures in psychiatry. The TRANSFORM-2 trial compared intranasal esketamine versus placebo, both started with a new antidepressant. The figure below shows each patient’s baseline depression rating scale score (on the MADRS) on the X-axis and the 28-day score on the Y-axis. Scores below the diagonal line show improvement from baseline, meaning that most people got much better with both treatments by day 28.
I’ll wager that you can’t tell which color is esketamine and which is placebo.4 This reflects the two fundamental facts about treatments for common psychiatric disorders. First, our treatments have small effect sizes that, in general, are much smaller than the overall improvement due to non-pharmacological factors (at least in clinical trials). Here, the esketamine arm improved by 21.4 MADRS points vs 17 with placebo augmentation.
Second, look at the range of outcomes. There’s a huge range of improvement for subjects receiving either placebo or esketamine. Subjects with baseline MADRS scores of 38 (severely depressed) ended up with final scores ranging from 0 to 39 with placebo augmentation(!). All of these patients met strict inclusion and exclusion criteria, and then received the same treatment in the same setting.
It’s humbling to see these data. If you only look at the figure from the publication below, you can still see a meaningful placebo response (the top dashed line), but hidden is the huge variability in outcomes. You would get the impression that the two outcomes are easy to tell apart, and that most outcomes clustered around that line. Neither of these is true.
Example 2: Quetiapine (Seroquel) monotherapy for bipolar II depression
The second example comes from a meta-analysis of four RCTs comparing quetiapine monotherapy (at two different doses) versus placebo for bipolar II depression. The figure below again shows outcomes for each patient based on their baseline and end-of-trial depression scores, just like the esketamine figure above.
Again, I’ll wager that you probably couldn’t identify the placebo arm without the legend, even though the effect sizes were larger for quetiapine (~0.45) than for antidepressants (~0.3) versus placebo. That’s because of the overlap in the outcomes of placebo and quetiapine, and the wide variation in outcomes.5
Example 3: Cariprazine augmentation for depression
Example 3 is from an individual-patient meta-analysis of two large 6-week cariprazine augmentation trials in depression. This is a spaghetti plot that shows the trajectory of each subject’s symptom improvement over time. The colors refer to various degrees of improvement by the 2-week mark and can be ignored. Panels A and B are the cariprazine augmentation arms (1.5mg and 3.0mg/d respectively) and C is placebo. D is the average improvement for each arm with 95% credible intervals.
I’ll let these figures stand by themselves. Look at the range of improvements, and how poorly the average score reflects this variation (panel D).
Example 4: Dextromethorphan-bupropion (Auvelity) for depression
The final example is the phase 3 GEMINI trial of dextromethorphan-bupropion, (Auvelity, originally called AXS-05) versus placebo. This is a spaghetti plot from the FDA review that shows the trajectory of each subject’s symptom improvement over time (the colored lines reflect dropouts due to different causes and can be ignored).
What do we make of this?
After looking at these trials, could you pick a random patient and know whether the pharmacological actions of the medication were responsible for the improvement? Absolutely not. At best, you could make an educated guess, but the probability that you are right is not great, as Hengartner and Plöderl describe for antidepressants:
When based on approximately normally-distributed interval scales, [the effect size] d = 0.3 indicates that, first, the outcome of antidepressants and placebo overlap by 88%, second, that only 62% of participants in the antidepressant group score above the mean of the placebo group and, conversely, 38% score below the mean (referred to as Cohen’s U3), and, third, that if you pick a person at random from the antidepressant group, he/she will have a minor chance of 58% to have the better outcome than a person picked at random from the placebo group (probability of 50% indicates no benefit at all). [emphasis mine]
None of these figures above are remotely controversial or disputed; they are just presented in a different manner from how we normally see them.6
If we can’t easily identify which patients got better against placebo, how can psychiatrists claim to be able to identify different responses between two active medications? I’m talking about situations where a psychiatrist is making a medication change and claiming that increasing to high doses of cariprazine is really effective (versus some alternative) or that they’ve had great success adding one medication rather than any of the alternatives they could have prescribed.
The simple reality is that, for common disorders like depression or anxiety, the variability in response and the size of the placebo response are both really large. This is why clinical trials need hundreds if not thousands of subjects to reach statistical significance. We’re not talking about the discovery of penicillin or the value of parachutes when skydiving.
Consider the cariprazine trials for depression which involved multiple RCTs with 750+ subjects each (as shown above), over 3000 subjects in total, and extremely strict exclusion criteria because the effects beyond placebo were so small—and even then they couldn’t detect a meaningful effect.7 And yet, psychiatrists will try to convince me that cariprazine has very distinct and detectable effects that differ from other treatments that they could have prescribed, or that it only works at certain doses that they can detect but the clinical trials couldn’t. I just don’t believe it.
Remember that each patient a psychiatrist treats is effectively a single-subject prospective, open-label observational trial where the control group is the memory of every other similar patient who the psychiatrist has seen. All of the elements of care that clinical trials try to control for—spontaneous improvement, placebo effects, beneficial contact with clinicians, comorbid diagnoses, co-prescription of other medications, substance use, severity, observation bias, and so forth—are missing. In highly controlled RCT conditions, you can barely detect an effect versus placebo in 3000 subjects, and psychiatrists claim to detect reliable treatment effects compared to other medications they would have prescribed?
An argument in favor of humility
I’m not arguing that we should completely disregard patterns in how patients respond to medications. I believe that careful clinicians can observe important trends in medications, particularly with adverse effects. Many medications have well-characterized effects, like antihistaminergic, antimuscarinic, antiadrenergic, or anti-dopaminergic (D2) effects. It’s clear that Seroquel (quetiapine) will cause more sedation and weight gain than Latuda (lurasidone), and Latuda will cause more akathisia.
What I am arguing for is humility: that noticing distinct patterns of improvement with psychiatric medications is really hard, and that we should be aware of our own biases and heuristics in the patterns that we see with patients. The brain is extraordinarily complicated and our knowledge of it is still limited. How the brain interacts with cultural values about psychiatry, illness, and medications is poorly understood. For the most part, we only have a limited understanding of why a patient got better after a medication change, and we have to sit with that discomfort.
The placebo researcher Ted Kaptchuk and colleagues framed this humility well when describing the outcome of a blinded trial for irritable bowel syndrome (IBS) where every patient received sham acupuncture. They describe the complex ways that people try to determine how a treatment is affecting them:
For most, treatment was a new and uncertain perturbation of an already complex life-world. The treatments provided the patients with an opportunity to reorganize that world, and their improvements mirrored the multilevel, pervasive role of IBS in their daily lives. Ambiguity often prevailed; certainty of improvement could be difficult to sustain. Our patients struggled with such questions as: Was it the therapy or the salad that produced the improvement? Was it real therapy or dummy treatment? and Was my digestion better or worse a week ago? Separating the placebo input from other aspects of daily life was not easy. Their testimonies has what Bruner (1986) called the “indeterminacy of reality” and “the trafficking in human possibilities rather than in settled certainties.”
The consequences of behaving as superstitious pigeons
In my experience, psychiatrist who rely heavily on their own experiences prescribing medications can practice in ways that are divorced from any evidence. Using extremely high doses. Complex polypharmacy with four or five medications. Adding a touch of Abilify or Caplyta (lumateperone) for everyone. Low dose naltrexone or low dose lamotrigine for a range of disorders. Red light therapy. Testing for esoteric vitamin and mineral deficiencies. Adding certain supplements, and even then only certain brands of certain supplements. Prescribing treatments based entirely on polymorphisms in COMT, MTHFR, or CACNA1C. I could go on endlessly.
And perhaps there are brilliant psychiatrists with decades of clinical expertise who pick up on subtle patterns the rest of us miss. Perhaps these psychiatrists can deviate from the evidence and practice creative courageous psychopharmacology. But not every psychiatrist is brilliant. Someone finishes at the bottom of the class. And when one psychiatrist insists they get amazing results with one treatment, I talk with another who assures me that a completely different strategy is actually effective.8 I routinely find that my anecdotal experience prescribing a medication differs completely from my colleagues.
To me, psychiatrists who claim to see reliable effects with very specific treatments are probably superstitious pigeons. Due to random variation, they had some patients get better with a treatment, this experience was reinforced by chance outcomes with other patients, and then they were biased to notice treatment successes and to disregard treatment failure due to their expectancy. I’ve seen this process happen as early as the first and second year of psychiatry residency, when trainees have strong beliefs about medications from only three or four experiences prescribing it (“BuSpar never works” or “risperidone is so much stronger than aripiprazole for psychosis”).
The deeper problem
The more abstracted harm is that psychiatrists and patients both grant far more power to biological effects than are warranted, according to clinical trials. If someone gets better with a new medication, the reality is that we don’t fully know why. And, if you take RCT data seriously, it’s typically driven by complex non-pharmacological factors for common conditions like depression (including expectancy related to the medication).
This uncertainty about outcomes extends to our own experiences with medications. None of us knows why we get better with a medication, whether it was purely the biological effect or non-pharmacological factors—even if we feel that it must be a pharmacological effect. If we imagine that we were a single patient in the esketamine or cariprazine trials above, how on earth could we know whether our improvement is due to active medication or other factors?
Consider qualitative studies of clinical trial subjects who received blinded placebo and refuse to believe that their improvement could have been due to placebo, as in this study of sham acupuncture for irritable bowel syndrome:
She was surprised (“Wow”) when the interviewer told her that she was in the placebo group. Finally, Frances’s big question was answered, and she was told that placebo needles did not pierce the skin. This directly contradicted Frances’s embodied experience that she had previously shared with the interviewer. The possibility that Frances was wrong was quickly dismissed—she was certain that she felt the needles penetrate her skin—and a collaborative effort was made to explore an alternative that was consistent with both Frances’s experience and the interviewer’s information, that a mistake had been made and some of Frances’s treatments had been placebo acupuncture and some had been real acupuncture.
In this case, it was so difficult to accept that the improvement the patient experienced was due to sham, that she convinced herself that the clinical trial made an error and gave her real acupuncture. I’ve seen something similar play out with many younger psychiatrists who have taken psychiatric medication themselves. Their experiences with medications powerfully informs their practice, and how could it not? But it is probably a mirage.
The consequence of all of this is that patients develop very strong narratives and beliefs about why they got better. I hear this from patients all of the time: that it just took the addition of 6mg of Vraylar to break through, or that a previous psychiatrist figured out the perfect cocktail of 5 different medications which reflects how complex the depression was. If we take the literature seriously, this is probably inaccurate. But it deeply shapes how people understand themselves.
The counterarguments
Clinical trials are not perfect, and there are many clinical scenarios where we lack high-quality evidence and need to rely on clinical judgement. Intelligent psychopharmacologists like David Healy and Ken Gillman have made strong critiques about flaws in psychiatric RCTs—the former focusing on issues of manipulated data in industry-sponsored trials and the failure of RCTs to detect adverse effects; the latter focusing on pharmacologically implausible outcomes, as with Seroquel having positive trials for too many disorders.
If you read my writing, I will be the first to say that many most positive industry-sponsored clinical trials should be accepted cautiously. Nevertheless, there are many high-quality trials in psychiatry that form a reliable evidence base. More often than not, these trials suggest that we should practice more conservatively than we do. We don’t have to disregard the entire scientific literature, when a more careful excision could do. Improving the quality of clinical trials seems to be the appropriate response to the limitations of our evidence in psychiatry, rather than abandoning them altogether. And what is the alternative to guide practice? We are already over-reliant on “I’ve had good luck with it.”
Final thoughts
I remember reading Peter Kramer’s Listening to Prozac before medical school. I was affected by the idea that a prescription could visibly change someone’s personality structure. I imagined that I would have philosophical discussions with patients about what it meant for a medication to have the capacity to change them. But as I get further into my clinical career, I am not so certain that Kramer got it right.
The more humbling, but more interesting, reality is that definitive medication effects are hard to pin down for most patients. My patients and I are left trying to sort out medication effects in the midst of complicated and messy lives. More often than not, I am the one questioning whether we should attribute so much causality to a medication, even if my patient wants to. While I view my role as helping someone with depression feel better, I increasingly feel a second responsibility to my patients: to help them recognize uncertainty in medication effects so that they can understand themselves in nuanced, complicated ways.
I first learned of this from my brilliant former program director Dr. Jonathan Heldt.
As in all of my writing, I use “psychiatrist” as a stand in for any clinician prescribing psychiatric medications: primary care physicians, nurse practitioners, and physician assistants.
I’ll focus on the treatment of common conditions like depression where non-pharmacological effects are large, but it could apply to much of psychiatry.
Red is antidepressant + placebo. Blue is antidepressant + esketamine.
For the nerdy: Mean change in MADRS total score from baseline to week 8 [SE]: −15.58 [0.62] and −14.88 [0.62] for quetiapine 300 and 600 mg/day, respectively, vs−11.61 [0.70] for placebo; LOCF; p < 0.001, both doses). Effect sizes, based on MADRS scores, for both quetiapine 300 and 600 mg/day: MMRM: 0.44 and 0.47, respectively, p < 0.001 vs placebo.
Let me emphasize that I am making a distinction between the pharmacological and non-pharmacological effects of a treatment. Non-pharmacological effects include factors like patient expectancy, beneficial contact with the clinician, natural fluctuations in symptom severity, and so forth. In practice, non-pharmacological effects would occur with any other treatment that could be prescribed, plus the effects of that medication. In the figures above, subjects got massively better with esketamine augmentation, quetiapine, cariprazine, and dextromethorphan-bupropion, but they also got massively better on placebo (a non-pharmacological response).
In fairness, the also may have issues with subjects enrolling for financial reasons or symptom rating scales being fudged by investigators so that a subject qualifies for enrollment. Randomization to placebo would then show a return to the lower score, and make placebo appear more effective.
And whenever I am told that psychiatrists deviate from clinical practice because they can identify real effects in real patients that are not detectable in the artifice and messiness of clinical trials, I remember all of the places where psychiatrists practice against the expert consensus: Antidepressant monotherapy in bipolar disorders. Subtherapeutic doses of prazosin. Uses of benzodiazepines in PTSD. Underuse of clozapine and excessive serial antipsychotic trails in schizophrenia. Excessively high antipsychotic doses. Antipsychotic polypharmacy. And so on. We see this pattern in the other medical specialties even when there are known concrete outcomes. Some doctors just aren’t very good!
Deep. Good. Scary. Been there, not done that.
But, most of my colleagues have. I would not call it superstition unless it worked more than once that turning my head in one direction would bring me food. The pigeons you describe, to become superstitious, would need to experience the coincidence more than a few times. I suppose pigeons may not need much proof. Most humans would need a bit more than that.
When docs are prescribing based on poor or hyped science or a need to be helpful, I would not call it superstitious as much as I would call it "the unexamined practice specified by some other "authority." That would be "sheep."
You can be a non-science major of just about any kind and apply to med school (or other similar training) as long as you pass certain "scientific" classwork. I am not up to speed on how much of that classwork, these days, would require significant intimate familiarity and hands-on experience with the scientific method and critical thinking in that realm, but I think (I could be wrong) not much.
Try not being "sheep" these days in your likely well-supervised practice, and the lesson may be painfully learned. I would think that following recipe books and error-ridden "legacy beliefs" is a way bigger problem than individual superstitions.
Late to the party on reading but just wanted to express my appreciation. It's a common way that older psychiatrist can dismiss the opinions of younger psychiatrist by saying that they have "more clinical experience" (a relevant comment for some aspects of practice for sure but not for others) it's great to see more conversations about this