Hey folks!

I’m excited to be firing up the newsletter and YouTube feed again for the academic year. I hope you all had a great summer and were able to get a lot done while enjoying the opportunity to rest a bit.

This summer I was able to finally resubmit and publish a manuscript I’ve been working on for a few years to rarefaction. This is now my third paper on rarefaction. Don’t worry, it’s the same story. If you are analyzing microbiome data, please use rarefaction.

This paper raised a few existential questions for me. At what point should a paper be retracted or corrected? If you take the time to read through this recent paper of mine, you’ll notice a lengthy supplemental text document where I went through some people’s code showing how it was basically flawed from the start. The code was being used to show that the tool being proposed worked well. Only, when I fixed the code, I showed that the tool didn’t work as well as advertised and certainly wasn’t nearly as good as rarefaction. I don’t think there was any fraud here, perhaps a good amount of sloppiness. Somewhat to my frustration, although the authors were given ample opportunity to reply to my paper, they didn’t.

Here’s the question: Should the original paper be retracted? Corrected? I really don’t know. The original paper has 424 citations and was published 7 years ago. I don’t know that any real damage has been done. If those 424 citations used the tool, I suspect they had an analysis that wasn’t as robust as it could have been. But this paper is still in the scientific record. I think we are willing to accept some amount of poorly done science in the literature that we can navigate around. This returns to my question: at what point is science so poorly done that it needs to be pulled?

This brings me to the set of figures I’ll be discussing on YouTube next week. Earlier this summer a paper titled, “Bacteroides fragilis toxin promotes gall bladder cancer in mice” was published in Nature Microbiology. They used shotgun metagenomics to quantify the abundance of different bacterial taxa on and off gall bladder cancer tumors in human patients. The claim was made that enterotoxigenic B. fragilis was differentially represented on and off of the tumors. They then explored this further using mouse and tissue culture models. Here’s part of Figure 1

I’ll level with you: the figures here are not good. “Expression”… as a measure of DNA abundance? Those relative abundance values are miniscule. Wait for Monday’s video for the rest of the critique.

Here’s the problem. Look at panel C. It’s showing fold change/difference of B. fragilis abundance on tumors vs. off tumors for the same patient. They have a significant p-value. But those fold change values are not normally distributed and many of the slopes are negative. If they had used a Wilcoxon test instead of a T-test, the P-value would have been much larger. It likely would have killed their study.

Here’s another problem. A lot of the species listed in panel A are not commonly human associated bacteria. A lot of them are things we might expect to show up in reagents. This is even more apparent if you look at Extended Data Figure 1. Bacteroides don’t even make a blip in this figure. Cyanobacteria? Pseudomonas? Why do patients 1-8 look so different from 9-25? Beyond the choice of how to present the data, the data raise many red flags.

This work is not unlike the placenta microbiome controversy. Just because you can detect DNA, does not mean you have live bacteria that resided in a niche.

In this Nature Microbiology article the authors went on to do a lot of other work that seems legit. But the foundation of the story is pretty soft. Based on these data, I don’t believe these patients had a disproportionate level of B. fragilis, much less enterotoxin producing B. fragilis, on their tumors relative to off their tumors. If I am correct, then the rest of the story loses a lot of its value. A bacterium that produces a toxin will do all sorts of damage regardless of where it is placed. But is it even there in humans?

So again, what is the correct response? I’ll post my YouTube videos going into greater detail with the problems. I suppose I can put a link to the videos on PubPeer. But do results like these mean we should write letters to the editor officially expressing concern? I can tell you that this type of work takes a lot of effort. If the editors are going to pan my comments like the authors did when I emailed asking for the data in Exploratory Data Figure 1 or for their sequencing data, then I’m not sure its worth my effort.

Why does this matter? Well, I don’t want slop to accumulate in the literature. Also, it grates on me to no end that people get this kind of work published in “high impact journals” while people in smaller labs struggle to published robust work in middling journals. I also fear that this type of work is about to explode in the age of generative AI where one can upload a table and ask the LLM to make a plot or run a statistical analysis that no one can reproduce. The fact that the authors described their results as “expression” in Figure 1a tells me they do not understand what they are analyzing. We need to figure out the answer to how to respond to the paper I did a detailed analysis of and how to respond to this paper so that when things get really hard we know how to respond. Ideas?