Statistical analysis, misused, allows you to prove many things that are not true. This was long a feature of advertising: with our toothpaste you get 38% fewer cavities, etc. In the past such ‘studies’ were not published in respectable journals, and research supported by on such was not funded. Now it is published and it is funded, and no one much cares. For an academic, this is the only game in town. One result, well known, is the “crisis of replicability”– very few studies in medicine, psychology, or environment are replicable (see here for more).

In this post, I look at food health claims– studies that find foods cause cancer, or cure it. The analysis I present comes from two researchers, Schoenfeld and Ioannides, (read the original article here) who looked at the twenty most common ingredients in “The Boston Cooking-School Cook Book”. For each food, they used Pub-Med to look up the ten most recent medical articles that included the phrase, “risk factors”, the word “cancer”, and the name of the food in the title or abstract. For studies finding effect in the range of 10x risk factors to 1/10 risk factors, the results are plotted below for each of the 20 foods. Some studies showed factors beyond the end of the chart, but the chart gives a sense. It seems that most every food causes or cures cancer, often to a fairly extreme extent.

A risk factor of 2 indicates that you double your chance of getting cancer if you eat this food. Buy contrast, as risk factor of 0.5 suggests that you halve your cancer risk. Some foods, like onion seem to reduce your chance of cancer to 1/10, though another study say 1/100th. This food is essentially a cancer cure, assuming you believe the study (I do not).

Only 19% of the studies found no statistically significant cancer effect of the particular food. The other 81% found that the food was significantly cancer-causing, or cancer preventing, generally of p=0.05 to 0.05. Between the many studies done, most foods did both. Some of these were meta studies (studies that combine other studies). These studies found slightly smaller average risk factors, but claimed more statistical significance in saying that the food caused or cured cancer.

The most common type of cancer caused is Gastrointestinal. The most common cancer cured is breast. Other cancers feature prominently, though: head, neck, genetilia-urinary, lung. The more cancers a researcher considers the higher the chance of showing significant effects from eating the food. If you look at ten cancers, each at the standard of one-tailed significance, you have a high chance of finding that one of these is cured or caused to the standard of p=0.05.

In each case the comparison was between a high-dose cohort and a low-dose cohort, but there was no consistency in determining the cut-offs for the cohort. Sometimes it was the top and bottom quartile, in others the quintile, in yet others the top 1/3 vs the bottom 1/3. Dose might be times eaten per week, or grams of food total. Having this flexibility increases a researcher’s chance of finding something. All of this is illegitimate, IMHO. I like to see a complete dose-response curve that shows an R² factor pf 90+% or so. To be believable, you need to combine this R² with a low p value, and demonstrate the same behaviors in men and woman. I showed this when looking at the curative properties of coffee. None of the food studies above did this.

Of course, better statistics will not protect you from outright lying, as with the decades long, faked work on the cause of Alzheimers. But the most remarkable part is how few people seem to care.

People want to see their favorite food or molecule as a poison or cure and will cite anything that says so. Irreplicable studies are cited at the same rate as replicated studies, as shown in this 2020 study by Yang Yang, Wu Youyou, and Brian Uzzi. We don’t stop prescribing bad heart medicines, or praising irreplaceable studies on foods. Does pomegranate juice really help? red wine? there was a study, but I doubt it replicated. We’ve repeatedly shown that aspirin helps your heart, but it isn’t prescribed much. Generally, we prefer more expensive blood thinners that may not help. Concerning the pandemic. It seems our lockdowns made things worse. We knew this two years ago, but kept doing it.

As Schoenfeld and Ioannides state: “Thousands of nutritional epidemiology studies are conducted and published annually in the quest to identify dietary factors that affect major health outcomes, including cancer risk. These studies influence dietary guidelines and at times public health policy… [However] Randomized trials have repeatedly failed to find treatment effects for nutrients in which observational studies had previously proposed strong associations.” My translation: take all these food studies with a grain of salt.

Robert Buxbaum, April 4, 2023