Scientific Nutrition Update 12: Beans and Heart Disease and Cancer and P Hacking

This is my single favorite episode of this podcast that I have ever made.  In it I discuss a recent study about beans, and use it to explain some of the problems with nutritional science.  Check this one out:

 

Study here

Script:

For today’s episode we are going to talk about the magical fruit, that’s right beans.  Now the reason we are discussing beans today is that a brand new study just came out, and using this will help us better understand some of the problems with nutritional science, and why it is so hard to analyze it objectively.

 

So brand new study published on legumes which are beans of all things.  I found it scrolling through twitter when Dr. Matthew Dalby tweeted about it, and discussed the results.  I was immediately interested because it draws some really interesting conclusions. In this study they basically took the data set from a different study that was looking at the mediterranean diet for heart disease and tried to analyze it to see the effects of beans.  In the original study people already at high risk of heart disease were recruited, and some were placed on the Mediterranean diet and their outcomes were compared to those who were not, specifically looking for a change in their likelihood of developing cardiovascular disease.  Are you struggling to see the connection to beans because I was too, but just wait it gets better.

 

In order to do their analysis they took this big pile of data, and decided they were going to divide into three smaller piles based on how many legumes/beans were consumed daily.

 

To make it better, there is less than a 1 oz difference per-day between the smallest group and the largest group in terms of legume consumption.  

 

Once they had all of their people in little piles, they went wild and ran every single statistical test known to man on it.  

 

So now you may be curious what results were found, first I’m going to drop the big one, there was no link between legume consumption and all-causes mortality.  Makes no difference. They then also concluded if you move up less than an ounce of beans a day from the lowest group to the highest, your risk of dying of a heart attack goes up by almost 50%!! This was truly shocking to me so I kept reading.  We all lucked out though because your risk of getting cancer goes down by almost 50%! This was fantastic news. No one wants cancer, so I looked at that data closely and started to get a little bit more angry. If you move from the lowest group to the highest group, you drop your risk by almost 50% but if you move from the lowest group to the middle group you increase it by almost 20%!  Now it doesn’t actually increase by 20% because the confidence intervals overlap, but I’m going to explain that in just one second.

 

So right now you may be wondering, well how in the world am I supposed to interpret this? Are beans good or bad? Protective or dangerous?  I’m going to give you the simple answer to this: they are good for you. High in fiber, high in protein, high in antioxidants, and inexpensive they are an incredibly useful staple ingredient for eating and cooking with.

Your follow-up question likely has to be about how I interpret this study, or why I seem to be getting progressively angrier the more I talk about it.  That one is going to need to give a little bit of explanation about what good science is and what some of these terms I am going to use mean.

 

First things first we need to discuss the data set they used, picking one like this where it was originally assembled for a specific intervention is already problematic, because it becomes difficult to stratify out our desired values.  If we instead were curious about the mediterranean diet and just wanted to see how different stratifications or disease states might affect it, I could see this data-set being a reasonable choice.

 

Next we need to discuss the three tertiles or piles they divided it into.  There is significantless than a 1 0z/day difference between the largest and the smallest group.  Very few food groups will have any affect at that small of an intervention size.

 

Now we need to briefly discuss how we determine if something is significant in science.  I am going to keep this as brief as I can, but we use a statistical tool called a P-value, now all a P-value is is a probability that says the likelihood your result was due to chance.  (This is not exactly true but the definition works for our purposes here). We commonly say that anything with a P-value of less than .05 is significant, so less than a 5% chance it is due to random fluctuations or chance.  The issue with this becomes something called P-hacking which has become endemic in nutritional science. Nutritional scientists are often under intense pressure to punish positive results, meaning times they found there to be a significant results between a certain foodstuff and certain diseases.  They are also increasingly gaining access to large data-sets and increasingly sophisticated computer-aided statistical analysis tools. This becomes problematic, because it has become easy for scientists to run a huge number of statistical analyses against a large data-set and eventually some correlations will pop up with p-values less than five.  

 

Reading this study, I am almost sure this is what was done here.  They took this large data set from the PREDIMED study and decided to run a whole bunch of statistical tests on it until something popped up that could be studied.  In this case a relationship between legumes and lower cancer risk and higher cardiovascular disease risk. Two big diseases with a lot of money behind them. So now they P-hacked their data, found an association and will now be better able to apply for grants that will allow them to better study the link between legumes and cancer and cardiovascular disease, despite the quite large likelihood that there is no association.  Remember a P-value is just a probability, so .05 still means 1/20 this is due to chance, and if you cut your data into enough slices you’ll be able to find that.

 

The other reason reading this that I immediately knew this study was dangerous was the size of the effect.  It is unheard of to hear of a less than 1 0z increase in consumption of a common and safe foodstuff to cause a 50% increase in cardiovascular disease or 50% decrease in cancer.  I knew just intuitively that there was no way 1 oz of beans was having that large of an effect on humans. It didn’t make fundamental biological or biochemical sense.

 

Now the reason this kind of study pisses me off so much, is because of the headlines.  I have a feeling that before I get a chance to publish this podcast, someone will have posted a headline saying Beans Cause Heart Attacks, and for the vast majority of people they won’t have the training to actually analyze the study and they will believe it.  But then they will go over to the other news site they read and read Beans Prevent Cancer, and again these studies are hard to analyze, and people will end up concluding that all nutritional claims are BS. This is dangerous for the state of nutritional science specifically, but all of science more generally.  That is why I am so strongly opposed to this practice.

 

If you want to find this study it is in Clinical Nutrition and is called Legume consumption and risk of all-cause, cardiovascular, and cancer mortality in the PREDIMED study.

 

If you find any great articles on the science of legumes or anything else you think I might find interesting email them to me at scinutrient@gmail.com  If you have any other questions send me a voice message on anchor and I’ll try to answer them on this podcast

 

If you want to learn more about intermittent fasting consider checking out my book The Optimized Guide to Intermittent Fasting and if you enjoyed this podcast, please subscribe and leave a review, it helps more than you know.  Thank you for tuning in, and remember live long, live healthy, but most of all live happy