Oh No Stanford No No No Not Again Please Make It Stop

Susie Neilson writes:

Tony Robbins was having a rough year. It was 2019 . . . Buzzfeed News was publishing a multi-part investigation into allegations that he had, during his in-person events, groped women and belittled abuse survivors. Robbins had a lot to lose. . . . he had amassed a net worth estimated at more than $400 million. His sweaty self-help seminars and grab-life-by-the-balls philosophy had spurred devotion from millions of followers, many of whom spent thousands on his products. . . . But at a precarious time, Robbins found an unlikely ally: Stanford.

No, not this guy. Not this person either, nor is it the hypester mentioned here. Not this group either. No, it was some other crew:

Around the same time that Buzzfeed News published its series, the Stanford Healthcare Innovation Lab, helmed by acclaimed genomicist Michael Snyder, launched a very different kind of investigation into Robbins’ seminars as part of an effort to identify “novel approaches to mental health.” In 2021, researchers affiliated with the lab, known as SHIL, published a study of “Unleash the Power Within,” Robbins’ four-day flagship seminar. . . .

Then, in 2022, SHIL-affiliated researchers — some of whom were fans and acolytes of Robbins’ work — published a more provocative paper. This one claimed that Robbins’ six-day, $4,500 “Date with Destiny” program eliminated symptoms of depression in 100% of initially depressed event-goers who were studied. In contrast, across clinical trials of antidepressants, just half of people report feeling better in six to eight weeks.

100% improved, huh? That sounds like big news! And, indeed, the Stanfordians were pretty proud, as well they should have been after such a clear and positive finding:

“This is going to be one of the most effective, if not the most effective, improvements in depression published,” Ariel Ganz, SHIL’s director of mental health innovation and the studies’ co-author, said in a 2021 video conversation with Snyder, other co-authors and Robbins.

In case you’re curious, the two papers in question are here and here

Sorry, now comes the bad news:

But when the Chronicle asked more than a dozen experts in psychology, statistics and medical research to review Stanford’s Date with Destiny study, many raised serious concerns about its validity. They found basic calculation errors, head-scratching data points and conflicting statements about how study participants were selected. Critically, they noted that too few people participated in the research for the findings to hold meaning for the public at large.

That’s a bummer when your study has data problems. Really too bad. That paper had 26 participants and 9 authors—that’s less than 3 data points per author, better than the student-faculty ratio at Ivy League schools. You’d think the authors could’ve avoided all these errors by divvying up the problem and looking carefully at the data from three participants each. Now they’re in the same category as that gremlins guy who approached the Platonic ideal of publishing a paper with more errors than data points.

The authors responded:

Snyder declined an interview, asking for reporters’ questions in writing. In response to a list of detailed questions, he acknowledged the study must now be corrected in light of the Chronicle’s findings, while noting that it was peer-reviewed. Snyder said that, in any event, the resulting paper suggested that “immersive interventions may be useful for reducing depressive symptoms and enhancing well-being.”

Ahhhh, the one-way street fallacy! Their data are also consistent with no effect, or with a negative effect.

Neilson continues:

Neither of the studies bore marks of research misconduct or fraud, according to the experts contacted by the Chronicle. Yet the work doesn’t appear to be of the caliber expected of a researcher with Snyder’s reputation, they said, or of a university such as Stanford. . . .

“Has he lost his mind?” asked Phyllis Gardner, Snyder’s colleague at Stanford’s School of Medicine, who was among the first to question the work of disgraced Theranos founder Elizabeth Holmes.

“That seems really surprising to me,” said Dr. Max Wintermark, the former deputy director of the Stanford Center for Precision Mental Health and Wellness, who now chairs the University of Texas MD Anderson Cancer Center’s neuroradiology department.

What was going on?? Here’s the story:

The Chronicle spent several months looking into Stanford’s unusual research. The partnership behind the studies formed when Ganz, then a new postdoctoral researcher in Snyder’s lab, met Benjamin Rolnik, a former Hollywood talent agent, at a wellness retreat hosted by one of Robbins’ close friends and collaborators, self-help guru Byron Katie.

Wow, this is a wonderful origin story. Much better than anything we have in statistics. The story continues:

Robbins and Snyder backed the same startup; they have also promoted each others’ products and business partners. . . . Robbins has also elevated controversial ideas and therapies. In the 1990s, he expressed doubt about the link between HIV and AIDS . . . Like Robbins, Katie has been criticized in online forums for belittling the experiences of abuse survivors during her own events. In 2012, she told an interviewer that she “never” felt sorry for people who experienced rape and other abuse, because “they only believe their thoughts” and were “perfectly all right.” In an email to the Chronicle, Katie said her statement had been “taken out of context.”

Wow. Also this:

Stanford did not respond to the Chronicle’s request for a list of donors to the genetics department and more information about SHIL members. . . . Under Snyder, Rolnik and Ganz went on to co-author four papers on Katie’s method, showing The Work could help stutterers, teachers and people whose genes put them at elevated risk for breast cancer.

And now some details:

James Heathers, a physiologist and affiliated researcher at Linnaeus University in Sweden, is best known for his work detecting errors in published research. . . . Heathers commended the Stanford researchers for publishing their data in a Dropbox folder, which he said is a good practice that demonstrates openness, but said he was left with broad concerns after reviewing it.

“Overall: This study is poorly conducted, and contains data handling errors, protocol violations, and other evidence of poor experimental practice,” Heathers wrote in a document summarizing his line-by-line review of the Date With Destiny study.

Critically, the researchers relied on patients’ PHQ-9 scores but calculated them incorrectly, Heathers noted. Instead of adding up each of the nine items, they added item No. 2 twice and skipped over item No. 3. These scores form the bedrock of findings about patients’ changing depression levels. “It is hard to imagine how this escaped inspection,” Heathers wrote.

OK, but this last bit is funny

I’ve tried to present the above story in an entertaining way, but ultimately it’s sad (people with health problems being led on by hype), infuriating (academics who are already rich and successful cutting corners to gain even more riches and fame . . . what is it they want, a jet ski made out of diamonds? Is there anything that would satisfy these people), and upsetting (as this takes up financial and attentional resources that could be spent on actual science).

But the story does have one funny part, and here it is:

Representatives for Elsevier, the company behind the Journal of Psychiatric Research, which published the study, declined to provide details about the article’s peer-review process, but said it “upholds the highest standards of quality and integrity across all journals.”

I have a horrible feeling they’re being honest there. This published paper with all these data problems and conflict-of-interest problems probably does match a standard of quality and integrity that occurs across Elsevier’s journals.

They’re not even pretending to be shocked that they published such a bad paper. You gotta hand it to them for being so brazen about it. Not.