Tag Archives: replication crisis

Every food causes cancer, and cures it, research shows.

Leave a reply

Statistical analysis, misused, allows you to prove many things that are not true. This was long a feature of advertising: with our toothpaste you get 38% fewer cavities, etc. In the past such ‘studies’ were not published in respectable journals, and research supported by on such was not funded. Now it is published and it is funded, and no one much cares. For an academic, this is the only game in town. One result, well known, is the “crisis of replicability”– very few studies in medicine, psychology, or environment are replicable (see here for more).

In this post, I look at food health claims– studies that find foods cause cancer, or cure it. The analysis I present comes from two researchers, Schoenfeld and Ioannides, (read the original article here) who looked at the twenty most common ingredients in “The Boston Cooking-School Cook Book”. For each food, they used Pub-Med to look up the ten most recent medical articles that included the phrase, “risk factors”, the word “cancer”, and the name of the food in the title or abstract. For studies finding effect in the range of 10x risk factors to 1/10 risk factors, the results are plotted below for each of the 20 foods. Some studies showed factors beyond the end of the chart, but the chart gives a sense. It seems that most every food causes or cures cancer, often to a fairly extreme extent.

Effect estimates by ingredient. From Schoenfeld and Ioannides. Is everything we eat associated with cancer? Am J. Clin. Nutrition 97 (2013) 127-34. (I was alerted to this by Dr. Jeremy Brown, here)

A risk factor of 2 indicates that you double your chance of getting cancer if you eat this food. Buy contrast, as risk factor of 0.5 suggests that you halve your cancer risk. Some foods, like onion seem to reduce your chance of cancer to 1/10, though another study say 1/100th. This food is essentially a cancer cure, assuming you believe the study (I do not).

Only 19% of the studies found no statistically significant cancer effect of the particular food. The other 81% found that the food was significantly cancer-causing, or cancer preventing, generally of p=0.05 to 0.05. Between the many studies done, most foods did both. Some of these were meta studies (studies that combine other studies). These studies found slightly smaller average risk factors, but claimed more statistical significance in saying that the food caused or cured cancer.

0.1 0.2. 0.5 1. 2 5 10
Relative risk

The most common type of cancer caused is Gastrointestinal. The most common cancer cured is breast. Other cancers feature prominently, though: head, neck, genetilia-urinary, lung. The more cancers a researcher considers the higher the chance of showing significant effects from eating the food. If you look at ten cancers, each at the standard of one-tailed significance, you have a high chance of finding that one of these is cured or caused to the standard of p=0.05.

In each case the comparison was between a high-dose cohort and a low-dose cohort, but there was no consistency in determining the cut-offs for the cohort. Sometimes it was the top and bottom quartile, in others the quintile, in yet others the top 1/3 vs the bottom 1/3. Dose might be times eaten per week, or grams of food total. Having this flexibility increases a researcher’s chance of finding something. All of this is illegitimate, IMHO. I like to see a complete dose-response curve that shows an R2 factor pf 90+% or so. To be believable, you need to combine this R2 with a low p value, and demonstrate the same behaviors in men and woman. I showed this when looking at the curative properties of coffee. None of the food studies above did this.

From Yang, Youyou and Uzzi, 2020. Studies that failed replication are cited as often as those that passed replication. Folks don’t care.

Of course, better statistics will not protect you from outright lying, as with the decades long, faked work on the cause of Alzheimers. But the most remarkable part is how few people seem to care.

People want to see their favorite food or molecule as a poison or cure and will cite anything that says so. Irreplicable studies are cited at the same rate as replicated studies, as shown in this 2020 study by Yang Yang, Wu Youyou, and Brian Uzzi. We don’t stop prescribing bad heart medicines, or praising irreplaceable studies on foods. Does pomegranate juice really help? red wine? there was a study, but I doubt it replicated. We’ve repeatedly shown that aspirin helps your heart, but it isn’t prescribed much. Generally, we prefer more expensive blood thinners that may not help. Concerning the pandemic. It seems our lockdowns made things worse. We knew this two years ago, but kept doing it.

As Schoenfeld and Ioannides state: “Thousands of nutritional epidemiology studies are conducted and published annually in the quest to identify dietary factors that affect major health outcomes, including cancer risk. These studies influence dietary guidelines and at times public health policy… [However] Randomized trials have repeatedly failed to find treatment effects for nutrients in which observational studies had previously proposed strong associations.” My translation: take all these food studies with a grain of salt.

Robert Buxbaum, April 4, 2023

Social science is irreproducible, drug tests nonreplicable, and stoves studies ignore confounders.

12 Replies

Efforts to replicate the results of the most prominent studies in health and social science have found them largely irreproducible with the worst replicability appearing in cancer drug research. The figure below, from “The Reproducibility Project in Cancer Biology, Errington et al. 2021, compares the reported effects in 50 cancer drug experiments from 23 papers with the results from repeated versions of the same experiments, looking at a total of 158 effects.

Graph comparing the original, published effect of a cancer drug with the replication effect. The units are whatever units were used in the original study, percent, or risk ratio, etc. From “Investigating the replicability of preclinical cancer biology,”
Timothy M Errington et al. Center for Open Science, United States; Stanford University, Dec 7, 2021, https://doi.org/10.7554/eLife.71601.

It’s seen that virtually none of the drugs are found to work the same as originally reported. Those below the dotted, horizontal line behaved the opposite in the replication studies. About half, those shown in pink, showed no significant effect. Of those that showed positive behavior as originally published, mostly they show about half the activity with two drugs that now appear to be far more active. A favorite web-site of mine, retraction watch, is filled with retractions of articles on these drugs.

The general lack of replicability has been called a crisis. It was first seen in the social sciences, e.g. the figure below from this article in Science, 2015. Psychology research is bad enough such that Nobel Laureate, Daniel Kahneman, came to disown most of the conclusions in his book, “Thinking, Fast and Slow“. The experiments that underly his major sections don’t replicate. Take, for example, social printing. Classic studies had claimed that, if you take a group of students and have them fill out surveys with words about the aged or the flag, they will then walk slower from the survey room or stand longer near a flag. All efforts to reproduce these studies have failed. We now think they are not true. The problem here is that much of education and social engineering is based on such studies. Public policy too. The lack of replicability throws doubt on much of what modern society thinks and does. We like to have experts we can trust; we now have experts we can’t.

From “Estimating the reproducibility of psychological science” Science, 2015. Social science replication is better than dance drug replication, about 35% of the classic social science studies replicate to some, reasonable extent.

Are gas stoves dangerous? This 2022 environmental study said they are, claiming with 95% confidence that they are responsible for 12.7% of childhood asthma. I doubt the study will be reproducible for reasons I’ll detail below, but for now it’s science, and it may soon be law.

Part of the replication problem is that researchers have been found to lie. They fudge data or eliminate undesirable results, some more some less, and a few are honest, but the journals don’t bother checking. Some researchers convince themselves that they are doing the world a favor, but many seem money-motivated. A foundational study on Alzheimers was faked outright. The authors doctored photos using photoshop, and used the fake results to justify approval of non-working, expensive drugs. The researchers got $1B in NIH funding too. I’d want to see the researchers jailed, long term: it’s grand larceny and a serious violation of trust.

Another cause of this replication crisis — one that particularly hurt Daniel Kahneman’s book — is that many social science researchers do statistically illegitimate studies on populations that are vastly too small to give reliable results. Then, they only publish the results they like. The graph of z-values shown below suggest this is common, at least in some journals, including “Personality and social psychology Bulletin”. The vast fraction of results at ≥95% confidence suggest that researchers don’t publish the 90-95% of their work that doesn’t fit the desired hypothesis. While there has been no detailed analysis of all the social science research, it’s clear that this method was used to show that GMO grains caused cancer. The researcher did many small studies, and only published the one study where GMOs appeared to cause cancer. I review the GMO study here.

From Ulrich Schimmack, ReplicationIndex.com, January, 2023, https://replicationindex.com/2023/01/08/which-social-psychologists-can-you-trust/. If you really want to get into this he is a great resource.

The chart at left shows Z-scores, were Z = ∆X √n/σ. A Z score above 1.93 generally indicates significance, p < .05. Notice that almost all the studies have Z scores just over 1.93 that is almost all the studies proved their hypothesis at 95% confidence. That makes it seem that the researchers were very lucky, near prescient. But it’s clear from the distribution that there were a lot of studies that done but never shown to the public. That is a lot of data that was thrown out, either by the researchers or by the publishers. If all data was published, you’d expect to see a bell curve. Instead the Z values are of a tiny bit of a bell curve, just the tail end. The implication is that these studies with Z= >1.93 suggest far less than 95% confidence. This then shows up in the results being only 25% reproducible. It’s been suggested that you should not throw out all the results in the journal, just look for Z-scores of 3.6 or more. That leaves you with the top 23%, and these should have a good chance of being reproducible. The top graph somewhat supports this, but it’s not that simple.

Another classic way to cook the books, as it were, and make irreproducible studies provide the results you seek is to ignore “confounders.” This leads to association – causation errors. As an example, it’s observed that people taking aspirin have more heart attacks than those who do not, but the confounder is that aspirin is prescribed to those with heart problems; the aspirin actually helps, but appears to hurt. In the case of stoves, it seems likely that poorer, sicker people own gas, and that they live in older, moldy homes, and cook more at home, frying onions, etc. These are confounders that the study to my reading ignores. They could easily be the reason that gas stove owners get more asthma toxins than the rich folks who own electric, induction stoves. If you confuse association, you seem to find that owning the wrong stove causes you to be poor and sick with a moldy home. I suspect that the stove study will not replicate if they correct for the confounders.

I’d like to recommend a book, hardly mathematical, “How to Lie with Statistics” by Darrell Huff ($8.99 on Amazon). I read it in high school. It gives you a sense of what to look out for. I should also mention Dr. Anthony Fauci. He has been going around to campuses saying we should have zero tolerance for those who deny science, particularly health science. Given that so much of health science research is nonreplicable, I’d recommend questioning all of it. Here is a classic clip from the 1973 movie, ‘Sleeper’, where a health food expert wakes up in 2173 to discover that health science has changed.

Robert Buxbaum , February 7, 2023.