How to Decode Nutrition Studies: Make Sense of Research Strength and Quality with Dr. Adrian Soto-Mota

Adrian: 
It’s important to be aware for what things a study is useful for and for what things it’s just isn’t. If your task is to cut paper but you bring out the hammer, you’re going to do an awful job. This doesn’t mean the hammer is a useless tool. It doesn’t mean you shouldn’t own a hammer. It doesn’t mean that all hammers is bad.

It’s just the wrong tool for the wrong task. The same happens with studies. There are some research questions for which NHANES is just fantastic, as good as it gets. However, there are some research questions for, and NHANES it’s just not good, and it likely never will be.

Bret: 
Welcome to the Metabolic Mind podcast. I’m your host, Dr. Bret Scher. Metabolic Mind is a nonprofit initiative of Baszucki Group where we’re providing information about the intersection of metabolic health and mental health and metabolic therapies such as nutritional ketosis as therapies for mental illness.

Thank you for joining us. Although our podcast is for informational purposes only and we aren’t giving medical advice, we hope you will learn from our content and it will help facilitate discussions with your healthcare providers to see if you could benefit from exploring the connection between metabolic and mental health.

One paper says a ketogenic diet reduces all cause mortality. Another paper says a ketogenic diet raises blood pressure. How do we interpret all this? Dr. Adrian Soto-Mota, a internal medicine clinician and teacher, a data scientist and a PhD researcher, joins us to dissect these trials a little bit more. But also how do we see them in the broader picture, including what he thinks epidemiology studies are really good for what they’re not good for. And maybe what tests we should be looking for instead for longevity and how we see longevity versus health. Great questions. Let’s hear from Dr. Adrian Soto-Mota.

Dr. Adrian Soto-Mota, thank you so much for joining me here at Metabolic Mind. 

Adrian: 
Thank you for having me. Always happy to talk about papers and about emerging evidence. 

Bret: 
Ah, of course. It’s wonderful to have you. So, that’s exactly what we’re gonna talk about today. When papers are published, they often hit the media cycle, and it’s hard to know how much to trust and how much to doubt.

And I think what frequently happens, if it agrees with our preconceived notions, we trust it. And if it doesn’t, then we doubt it. But that’s probably not the right way to do it, would be my guess. So I want to bring up two papers that really highlight this issue and have you go through your thought process as a data scientist, as a researcher, as a clinician.

Wearing so many hats, I think you are perfectly suited to help us better understand how to interpret these papers. So the first one was published in the International Journal of Cardiology and Cardiovascular Risk and Prevention, and the title is Ketogenic diets are associated with an elevated risk of hypertension insights from a cross-sectional analysis of the NHANES 2007 to 2018.

So, a lot of people think keto diet, a lot of fat, probably dangerous, right? There’s a preconceived notion there. So I think a lot of people, clinicians included, will read this title and say, ah, see, I knew it. The keto diet’s bad for you and is gonna raise your blood pressure. But give us your analysis and how you would interpret this and communicate it to the general public.

Adrian: 
Sure. first of all, I want to say that it’s not only hard to know when to trust or what to make out of the results of the study, it’s also very frustrating. And it’s one of the reasons why in recent days people tend to be skeptic of experts, you could say. They tend to be like how can it be possible studies are pointing out in different directions?

When I saw this study, one of the first things I thought was, and this is because I know the NHANES dataset myself, I’ve published studies with the NHANES dataset myself. One of the first ideas was okay, this is very problematic because we don’t know, actually, we know people in the NHANES dataset don’t know frequently which diet they’re actually eating.

And this comes back to another recent study published a few months ago estimating that it’s less than 4% of participants in NHANES who say they follow a low carb diet, actually follow a low carb diet. Less than 4% of patients in the NHANES dataset who think they’re following a low fat diet, they’re actually following a low fat diet.

So this is also very problematic. 

Bret: 
Yeah, and if I could interrupt for one second because this is such an important point. I’m glad you’re making this point. And just take 30 seconds to say what is the NHANES data set so people know what we’re talking about to start with. 

Adrian: 
Yeah. Dataset is, it’s datasets, because gladly there are many of them. They have been updating these national health surveys over time. It’s one of the best available worldwide, projects estimating people’s nutrition and dietary habits and health status and health outcomes. So it’s a nationally representative set of studies that have some variation in viable definitions or variable encoding over time.

But it’s also one of the oldest. It has been going on for quite a while now, and it’s incredible for research because again, it has been running for a while now. The dataset is publicly available and it has a very well curated data, variable dictionaries, and it’s something that lots of people know and trust.

However, not everything in NHANES, I don’t want to say, it’s not rock solid. The thing is that because of the nature of some of the things that are measured in NHANES, it’s not the same to see someone’s, person’s height in NHANES versus their dietary habits because height is a very objective measurement.

Blood glucose is a very objective measurement, and triglyceride is a very objective measurement. Whereas, what did you eat yesterday is not a very objective measurement, and this is why it becomes, not because it’s in NHANES means it’s rock solid data point. This is no one’s fault in the NHANES research team.

They’re all fantastic. It’s just the nature of the data of whatever they are measuring in different variables. So in this particular case 

Bret: 
It’s still food frequency questionnaires that rely on the person to remember a year ago, 10 years ago, whatever the case may be, because they’re infrequent analysis.

So subject to all the downfalls of that type of data collection. 

Adrian: 
Absolutely. Absolutely. So again, it’s just the nature of the data. And because many researchers are aware of this fact, now there is, you could say, meta research on studies on how much can we trust some set of points or results from the same study, and that’s why. 

Bret: 
So you were mentioning the study. You’re mentioning the study where people would say, yes, I’m following a low carb diet. But then when they actually looked at what they were eating, sorry, you said what? Only 4% of those were actually falling a low carb diet? That’s remarkable. 

Adrian: 
Yes, absolutely.

Bret: 
That’s remarkably bad. 

Adrian: 
Absolutely, but then again, that’s the nature of dietary recalls. And I’m not saying they’re useless. They are not. It’s just something that people need to bear in mind. So when I saw the headlines and then when I went through the paper, the very first pushback I had against the conclusions was, we already know that the source, the main source

for what would be the intervention or the exposure in this particular dataset, is not very trustworthy. So that was one of the main reasons why I didn’t take the conclusions at face value. That’s one of the reasons why I said, yeah, okay, let’s look at it. But it’s not likely that we can absolutely buy into what the researchers concluded. 

Bret: 
Yeah. So they, I guess they tried to get past that a little bit by assigning a dietary ketogenic ratio or what they call a DKR. And this is the first time I saw this type of complicated calculation to estimate how much someone is in ketosis or not.

But I guess the problem is if you’re starting with questionable data to begin with, then any formula you plug it into is subject to the same concerns, is it? Would you agree with that? 

Adrian: 
Absolutely, and it’s, I think it’s okay when researchers want to take into account that the variable they’re studying is not perfect and they try to design or to develop a method to make it a bit more trustworthy.

And this is something that has been done multiple times, and I take it as good effort from research team. Okay, we are aware this is not entirely trustworthy.

Let’s just make, walk an extra mile to make it a bit more reliable. And this is something that many researchers do. This is something I’ve done trying to get a calculation or something, changes in the same direction to then try to come up with an analysis that bolsters the method or the conclusions that you are doing. In one of the studies in the DIETFITS analysis paper, what we did to provide a more objective measure of adherence to a given diet because in SIETFITS there was a low fat group and a low carb group. We know that the triglycerides to HDL ratio changes when people restrict carbohydrates, and we know that their total cholesterol changes when they restrict fat. So, we use those biomarkers because, it’s perhaps worth emphasizing with the exception of the ketogenic diet, no other diet intervention has an adherence metric. So it’s, when someone tells you they are following a vegan diet, you basically trust them because we don’t have good biomarkers and certainly we don’t go with a camera, we don’t follow them to record exactly what they are eating.

It’s very hard to come with these biomarkers of adherence, and I think it’s a good idea to try to come up with additional measures for dietary adherence. An additional point that’s perhaps worth highlighting there is that when you come up with something like this, you allow for other researchers to use the same tool and either validate or correct your results.

So, the fact that you come up with a ratio or that you use something that other people are already familiar with allows others to see in a different set of data if it’s a reliable or unreliable way of measuring othering to the intervention you are interested in.

So I think it’s something new. I did not know about this ratio and I take it as a hopefully promising way of estimating better if someone is in a ketogenic diet or not in future studies. Because in my group, we do studies with a ketogenic diet. Then we can use these ratios to see if they correlate, with what we know is happening, when someone is reupping carbohydrates.

So, I think this is also, this is also part of the value that a study can have even if it comes from an imperfect set of data. 

Bret: 
Yeah, but I think that last part is so important to validate it against what we know happens with a ketogenic diet. Because like you said, you could, that the ketogenic diet is the one diet that you can test adherence with a beta hydroxybutyrate with some ketone measurement.

But this is, to my knowledge, has not been validated that way. So it’s 0.9 times the grams of fat plus 0.46 times the grams of protein divided by 0.1 times the grams of fat plus 0.58 times the grams of protein plus the grams of net carbs, which is a very complicated formula and still subject to the patient trying to remember how much, how many grams of protein and how many grams of carbs.

So, it’s getting thinner and thinner until how reliable it is. But that’s my question. Has this been tested to validate when this ratio is high that yes, the person has ketones in their blood and they’re in ketosis? But I think that’s what you were getting at, but I don’t believe that’s been done.

Adrian: 
No, I, don’t think it has been done either. \However, we do have studies, and in my case in my group, we have access to patients in which we are absolutely sure they are on a ketogenic diet and we have their blood samples. And we, in some studies, we provided their foods and in some others, we didn’t. But we have other ways of measuring or to assess that we have pictures or we have something else.

We can test or adjust these parameters. I would add that the formula you mentioned also assumes that the impact of protein, carbs, and fat on ketogenesis is constant. You don’t have a more complicated model, which with the physiology we know we have the intuition that we perhaps, you know what was 0.9 sometimes it’s 0.7 and sometimes it’s 0.95.

Or we know physiology is dynamic. So I will also say that despite having lots of inputs, it’s still a very simple model trying to explain something that’s dynamic. 

Bret: 
Yeah, very good point. And so then I guess a next step is when you have a paper like this that has reached a conclusion, but with some questionable methods and reliability of the data, you have to say, how does this fit into other studies maybe of a higher quality that we know? How did you put this compared to other studies that you’re aware of looking at ketogenic interventions for blood pressure?

Adrian: 
That’s the second, I think, that came to my mind when I was going with these, when I was reviewing the paper, that we do have randomized control trials, specifically designed to look at the effect of a ketogenic diet on blood pressure that compares a ketogenic diet against the most studied diet intervention for blood pressure, which is the DASH diet. And the ketogenic diet had a larger effect, a larger blood pressure reduction effect, than the DASH diet. We know, and we have better detail already on what happens to someone’s blood pressure when they go on the ketogenic diet.

So, I think that this also needs to be taken into account. I don’t remember if this study was cited or not in the discussion of the NHANES paper, but it certainly worth recalling that we have data from randomized control trials on this particular outcome with more reliable assumptions on what people were eating.

And the outcome was that instead of increasing blood pressure, it reduced blood pressure. 

Bret: 
All right, yeah. So, when it comes to those two types of studies, it’s clear data in one is a little more reliable than the other. But now let’s talk about the other paper because we’re talking about blood pressure, something that you can measure right away that’s going to change quickly.

But then there’s this question of who lives, who dies, who’s gonna die prematurely, who’s going to die of heart disease? And that’s really hard to test in a randomized controlled trial because it may take 10 or 15 plus years to get those outcomes. So, the same group published a paper, The ketogenic diet has the potential to decrease all-cause mortality without a concomitant increase in cardiovascular-related mortality.

That was published in Nature Scientific Reports. So here, people in the keto community were thrilled, i’m sure, as you see this headline that see ketogenic diet reduces all cause mortality. I knew it, but still some concerns about it because it’s subject to a lot of the same data questions, right?

Adrian: 
Absolutely, perhaps it’s something very human to have confirmation bias, right? if we try to, it’s easier to accept something that confirms what we already do or what we already think. However, I mean in these both case, in these both studies, we have pretty much the same dataset.

Both are NHANES and actually, and of course they are subject to the same limitations that we already discussed about NHANES, they actually included the same years. One goes from 2001 to 2018 and the other one goes from 2007 to 2018. So, we are sure that some of the same data points that are in one study are in the other study.

And, of course, it can be very frustrating, like how can two studies on the same datasets including some of the same years yield different conclusions? So not because we like the conclusions or the results in one study means we can forget the incurrent limitations we find in the other.

And this study, it’s the one that ha has this DKR. You could try to, depend on why it’s a better study in terms of it has more years it has, if you could say this complimentary analysis. But it’s again the same dataset with the same limitations we’ve already discussed.

So it really shouldn’t make us, it really shouldn’t make us conclude or draw a hard line because of the same reasons we can conclude the draw a hard line on the other study. Now, here it’s very important to mention that in contrast with the other case with the case of hypertension, we’ll never have a randomized controlled trial of a ketogenic diet versus another diet that lasts an entire life.

So that will never happen in humans, has been done in some annual models, but that will never happen in humans. So, also, to be fair and realistic, these studies have limitations. Sure, they are not enough to allow us to make our minds, but they may get as close as we can to an answer on that particular topic because NHANES, again it’s one of the best designed and executed national service, and we will never have lifelong randomized control trial in a particular comparing to diets.

So, I think that it’s important to be aware, that it’s important to be aware for what things a study is useful for and for what things it’s just isn’t. And an analogy I frequently use when I am teaching about research methods, or actually I use it as well when I am teaching about pharmacology, is not because if your task is to cut paper, but you bring out the hammer, you’re going to do an awful job. Maybe you can get clever on how to force it to make it work, but it’s going to look awful. This doesn’t mean the hammer is a useless tool. It doesn’t mean you shouldn’t own the hammer. It doesn’t mean that all hammers is bad.

It’s just the wrong tool for the wrong task. The same happens with studies. There are some research questions for which NHANES is just fantastic. As good as it gets. However, there are some research questions for, and NHANES it’s just not good and it likely never will be. So, this doesn’t say anything else beyond that.

Bret: 
And as you very well pointed out, we don’t have the randomized control trial. So you say, how about we put this compared to other similar studies and compare the findings? And we often see nutrition epidemiology studies define low carb and say that there’s a higher risk of all cause mortality in cardiovascular disease.

And I think that’s what drew so much attention that this is the opposite. But, of course, those studies frequently separate people into quartiles of carbs. So, the low carb is less than 37% of calories, which could be 200 grams of carbohydrate, so very different. Do you see this as a better dataset based on that or just still so flawed that it’s hard to say better or worse? 

Adrian: 
Yeah, I think it’s better than some of the other things that are already in the literature, which is perhaps why I would publish it as well. Like it’s not, if I were the reviewer, if I were the editor, I would’ve published it as well because I do believe it’s adding value to what’s already out there.

I understand it can be frustrating or it can be completely that, oh, this is bringing only more noise. Not necessarily noise as people call it. It’s inevitable in one way. it’s still adding value. I do see it’s better than some of the other things we have seen before.

In terms of the length of the US included the tools they’re using, there are some good methodological assets to it. However, we also need to be aware that getting hard conclusions from observational data, it’s just very hard.

It’s the exception and not the rule. And I know, and there’s a lot of hype about mental organization, which is perhaps a good example on how an observational study can be better sometimes but still not enough.

This is perhaps off topic, but some people have taken general randomization studies as lifelong randomized controlled trials. They are far from it, and they will never be, and not all general randomization studies are equally reliable. So I think that it’s just a reality. We just need to be aware of the reality check that observational data will only get us to a certain point and we just cannot squeeze it much more than that.

Is this a better squeeze of observational data? Yes, in some reasons, yes. But it’s, there’s still a limited amount of insight you can get out of a given method. 

Bret: 
Yes, it’s a great point. You mentioned the Mendelian randomization trials and, or studies, and the assumption is that one thing changed and so you can track because that’s the only thing that changed between these groups.

So, you know the answer, but in reality that’s not the case. Multiple things change. You’re just measuring the one thing so it doesn’t completely answer the question. But I want to go back to one other thing you mentioned. You said there are some things that the NHANES dataset is really good for. So I’m curious what, what’s an example of that because we keep hearing how bad these studies are and how it’s, they’re unreliable? And I’m frequently singing that song because the way it’s used for nutrition tends to be pretty poor, I think. But so what’s an example of something that’s really good for, so you can feel better about it?

Adrian: 
Everything that includes, for example, objective lab measurements because then we don’t rely on someone’s memory. So, if your question involves blood biomarkers that require large numbers, then you can say those are fantastic. For example, I think it’s also very useful national or representative studies in general are very useful for informing policy and many other countries, that’s why many other countries have nationally representative service.

For example, I’ve been collaborating, I’m part of the NCD risk collaboration. NCD stands for Non-Communicable Disease Risk Collaboration, which is an epidemiological project carried out by Imperial College researchers, Imperial College.

And, for example, many of the results in this collaboration are looking into what’s the trend of different comorbidities or different diseases in the last decades. So, for this particular research question, which is very useful for informing and designing policy, these service are just fantastic.

If you are interested in estimating, if obesity, the prevalence of obesity is growing, it’s decreasing, it’s changing over time. If its coexistence with other comorbidities is changing over time, then these studies are fantastic. If some of that questions involve lab metrics, it or blood biomarkers, these studies are fantastic.

And, again, it really depends on the research question. 

Bret: 
Yeah, that makes a lot of sense. Then, so we’re faced with the dilemma then of how are we going to predict risk on all-cause mortality risk on cardiovascular disease if we can’t do the nutritional randomized control trial for decades or more.

So, instead we have to follow surrogate markers or markers that may help us predict that risk. But often we may not agree with what the best markers are. So, if we’re talking cardiovascular disease and all-cause mortality, what do you feel are the markers or tests we should be focusing on in our studies?

Adrian:
I think this is an emerging research field in its own, and I am very optimistic of biological clocks. So, for example, we know that not everyone, we are given age, not all people with the same age are equally healthy. Some people are 60 years old and are incredibly fit and healthy, and some people are 40 years old and they are already suffering with many age related comorbidities.

So I think that, this is crucial. And for example, one of the research projects I worked a few months ago with one of my students was trying to implement the insight from these biological clocks in other medical calculators. Because, for example, let me make up a small parenthesis here.

Critical care patients are assessed their mortality, their short-term mortality risk daily. So, if you are in an ICU, every day someone will try to track with a mortality calculator for critical care patients how well you’re doing. Many of these calculators asked for someone’s age, however, they are assuming that everybody with the same age looks the same.

So, we knew this was not the case. And some people have been developing these biological age calculators either based in DNA methylation or some are based in more easily available blood biomarkers. There are an emerging tool and some things about them are very promising. And I think that changes in these biological age markers would be relevant or will be relevant.

There’s already a handful of studies showing how some interventions actually can even lower the biological age of someone, which gives you, which is fantastic. It allows you from a data analysis point of view to have the possibility of changing in the other direction. Whereas with chronological age, there is only changing in one direction.

So, now I think that’s, may seem trivial but the field of biological clocks is rapidly growing. It’s rapidly giving promising results. We are learning what are the best ways to use them, and that’s definitely something to keep our eyes on for the new future at the same time, because the main reason the main mortality or the killer number one in the world is still cardiovascular disease.

I think it’s very relevant to measure cardiovascular disease, and it’s free. You can get estimations from well-studied, some better designed than others, calculators and online tools for predicting someone’s risk of having a heart attack or a major cardiovascular event within the next 10 years, within the next five years.

And some of these calculator calculators can make adjustments or different, or ponder different sources of influence such as ethnicity, for example, or country in which a patient is living. Some of them are more complex. However, I think that, let’s remember that not everybody dies of a heart attack. And let’s also remember that longevity, it’s perhaps not the best goal we can go after.

One of my teachers during med school used to say that technically speaking, life is a sexually transmitted disease. It’s chronic, it’s degenerative, and it has a lethality of 100%. You could say, that’s true. So the goal is not to avoid the death, all of us are walking towards it.

The idea is to, as I like to tell my residents, the idea is to die young as late as possible. So, that’s why it’s also very important to measure quality of life. And that’s why it’s also important to measure someone’s capacity to carry out their choice or someone’s capacity to take care of themselves.

And that’s why these other metrics are also important because, not being cynical about death, if you have a study with a century of follow-up, everybody would end up then. So, I think that’s, we need to be as, in terms of analyzing biomedical data to be perhaps a little bit less obsessed with death and more obsessed or more focused on quality of life and these other very relevant metrics.

Bret: 
Yeah, I’ve gotta say to die young as late as possible sounds a lot better than life as a sexually transmitted disease. Those sound very different to me. I’m much more in favor of the way you phrased it. 

Adrian: 
I am as well. 

Bret: 
So, thank you, I appreciate that. I think this was a great discussion about these specific studies, but also how to interpret studies in general and how they fit into the general interpretation of what we read and what we find in science.

And you’re in such a great position to help us with that as a clinician, as an educator, as a data scientist, as a researcher. So, if people want to learn more about you and the work you’re doing, where would, where can they go to find out more? 

Adrian: 
I am. Yeah, I’m mostly active in X or Twitter, however you call it these days.

And whenever I disseminate my research, is usually there. And it’s at AdrianSotoMota, everything together. And yeah, that’s, that would be the place. 

Bret: 
Great. Thank you so much for taking the time to join us, and I look forward to talking to you again in the future. Thank you.

I want to take a brief moment to let our practitioners know about a couple of fantastic free CME courses developed in partnership with Baszucki Group by Dr. Georgia Ede and Dr. Chris Palmer. Both of these free CME sessions provide excellent insight on incorporating metabolic therapies for mental illness into your practice. They’re approved for a MA category one credits, CNE nursing credit hours, and continuing education credits for psychologists, and they’re completely free of charge on mycme.com. There’s a link in the description. I highly recommend you check them both out.

Thanks for listening to the Metabolic Mind Podcast. If you found this episode helpful, please leave a rating and comment as we’d love to hear from you. And please click the subscribe button so you won’t miss any of our future episodes.

And you can see full video episodes on our YouTube page at Metabolic Mind. Lastly, if you know someone who may benefit from this information, please share it as our goal is to spread this information to help as many people as possible. Thanks again for listening, and we’ll see you here next time at The Metabolic Mind Podcast.