So there’s another study out that supposedly shows eating read meat will kill you. Specifically:
Red meat consumption is associated with an increased risk of total, CVD,[cardiovascular disease] and cancer mortality. Substitution of other healthy protein sources for red meat is associated with a lower mortality risk.
I’ll save you some time and admit that, because I think their conclusion is wrong, I’m assuming there are problems in the study design and methodology. Let’s look at it and see if I’m right.
What did they actually study?
We prospectively observed 37 698 men from the Health Professionals Follow-up Study[HPFS] (1986-2008) and 83 644 women from the Nurses’ Health Study[NHS] (1980-2008) who were free of cardiovascular disease (CVD) and cancer at baseline. Diet was assessed by validated food frequency questionnaires and updated every 4 years.
Right there I can stop reading. Do you know what a “validated food frequency questionnaire” is? Once a year they would ask people, “Over the past year, how often did you eat each of the following items?” Seriously, that’s what they base these studies on.1
I can’t tell you what I ate last month, there’s no way I can give you a reasonable average over the past year. But for any study that uses data from the HPFS or NHS studies, that’s all they’ve got.
It’s not that bad is it?
Yeah, it is. Here’s their own support for using this data:
The reproducibility and validity of these FFQs have been described in detail elsewhere.9–10
So they’re referencing two other articles. That’s cool, we should build on the work of others. It doesn’t make sense to keep reinventing the same basic research.
But what do those two references actually say about food frequency questionnaires? From the first one:
Results: Using factor analysis, we identified 2 major eating patterns, which were qualitatively similar across the 2 FFQs and the diet records. The first factor, the prudent dietary pattern, was characterized by a high intake of vegetables, fruit, legumes, whole grains, and fish and other seafood, whereas the second factor, the Western pattern, was characterized by a high intake of processed meat, red meat, butter, high-fat dairy products, eggs, and refined grains.
…
Conclusions: These data indicate reasonable reproducibility and validity of the major dietary patterns defined by factor analysis with data from an FFQ.
In other words, these questionnaires are good for identifying whether people eat a “Western” diet or a “prudent” diet.2 These “diet patterns” are collections of multiple distinct variables, otherwise known as confounders. The correlation of these factors …
No, wait, I’m talking like a scientist. Let me say that more plainly: There’s no way in hell you can use that data to say anything about just red meat.
How about that second reference?
Mean daily amounts of each food calculated by the questionnaire and by the dietary record were also compared; the observed differences suggested that responses to the questionnaire tended to over-represent socially desirable foods.
Translation? People. Lie.
We all know what we’re “supposed to” eat. No one likes following the food pyramid, but we like saying we eat that way.
Why does this matter?
Some people like to do the right thing. They’re the ones who are going to “over-represent socially desirable foods”. They’re also the ones who are more likely to work out occasionally. Less likely to drink heavily. More likely to skip dessert. Less likely to smoke.
In other words, they’re more likely to avoid all the known risk factors. And as long as most of what they do is good for them, they’ll be healthier than people who do all the “bad” stuff — known to doctors as non-adherers.
Tom Naughton once described how this process could work to turn celery into diet enemy number one. Tell people celery is bad for them. All the adherers stop eating it. Twelve years later, the only people still eating celery are the ones who also smoke, drink, and sit on the couch eating pizza and wings.
And I can already guarantee the result: people who eat a lot of celery tend to die younger. This would, of course, prove that celery is a health hazard, right?
Of course not. All it would prove is that health-conscious people had heeded the warnings and were dutifully avoiding celery. Or, to look at it another way, it would prove that people who choose to ignore the dire warnings about celery are what doctors call non-adherers … or what I call people who don’t give a @#$%. It would prove absolutely nothing about the actual health effects of celery.
If the warnings about red meat are wrong, but all the people who care about their health stop eating it, who would be left eating red meat? That’s right, people (like me) who think the warnings are wrong; and people who don’t give a @#$%.
But just for the sake of argument …
(Damn, I sure do seem to say that a lot when talking about food survey studies.)
Let’s suppose the food surveys are accurate. How big a difference did they say it makes that you eat red meat? You can read the dry statistical analysis …
We estimated the associations of substituting 1 serving of an alternative food for red meat with mortality by including both as continuous variables in the same multivariate model, which also contained nondietary covariates and total energy intake. The difference in their β coefficients and in their own variances and covariance were used to estimate the hazard ratios (HRs) and 95% CIs for the substitution associations.4
And on and on and oh dear God that is boring! Or for a good summary we can turn to a comment when the study was reported in the Sydney Morning Herald:
The death rate of people under 40 years in the US is about 150 per 100,000 (and that’s all deaths not just health related). So according to the article about 30 of these people died because of meat consumption. Or 120 per 100,000 instead of 150 per 100,000. An difference of 0.03% in the death rate. But that sounds so much less exciting than 20% chance …
That’s what all this comes down to. If we all give up red meat, the death rate might go down by 0.03%. Even if the study proved that (it didn’t) is it really worth giving up steak for that?
And what about how they define red meat?
How is this even a question, right? Red meat is just … red meat, right?
Of course not. First, did you know pork is a red meat? The National Pork Board might disagree with that, but that’s what this study said. Did all the survey respondents know that?
And they didn’t distinguish between lean cuts or fatty, just whether it was “processed” or not. So it was a hot dog or it was a steak. Did the fat content matter at all? Who knows?
So of course they didn’t ask if it was grass-fed or grain-fed, even though the nutrition profile of the two kinds of beef is pretty radically different. Short version: Cows aren’t designed to eat corn, but we feed it to them anyway, and load them up on antibiotics to keep them from dying.
Once we finally decide exactly what they mean by “red meat”, that still leaves the question of …
What did they replace it with?
If I suddenly stopped eating red meat, either I’d start eating a whole lot of something else in its place or I’d be really freaking hungry. So for all these non-meat eaters, what were they eating instead? Somehow I doubt they all ate the same “something else”.
And finally, back to …
The statistical analysis
Remember that paragraph back near the middle where I pointed out how boring the statistics were? I suppose I should go ahead and critique their analysis and look for flaws.
Except no, I shouldn’t. Why? Because if you ask the wrong questions, the answers don’t matter:3
A Type 3 error “is the unintentional error of solving the wrong problems precisely.”
“In sharp contrast, the Type 4 Error is the intentional error of solving the wrong problems.”
- Type 3 errors are mistakes
- Good information sometimes fixes mistakes
- Type 4 errors are tend to be done for two overlapping reasons:
- There is money in solving the wrong problem.
- Someone is making money/getting elected keeping people from thinking about the right question.
- People making Type 4 errors fight correct information and resist change.
That’s the great shortcut to evaluating nearly every nutritional study you see reported in the press. Are they asking the wrong questions? And does it look like a Type 4 error? If the answers are “Yes” and “Yes”, you don’t even need to bother with the boring statistical part.
1) By the end they were doing the food survey once every four years. I swear I am not making this up.
2) The word “prudent” isn’t being used as an adjective. That’s “The Prudent Diet”, like “The Zone Diet” or “The Biggest Loser Diet”. So what is The Prudent Diet?
This theory [the lipid hypotheses] was tested in 1957 when Dr. Norman Jolliffe, Director of the Nutrition Bureau of the New York Health Department, initiated the Anti-Coronary Club. With great media fanfare, a group of businessmen, ranging in age from 40 to 59 years, were placed on the so-called Prudent Diet. Prudent Dieters used corn oil and margarine instead of butter, cold breakfast cereals instead of eggs and chicken and fish instead of beef. Anti-Coronary Club members were to be compared with a “matched” group of the same age who ate eggs for breakfast and had meat three times a day. Jolliffe, an overweight diabetic confined to a wheel chair, was confident that the Prudent Diet would save lives, including his own.
The results of Dr. Jolliffe’s Anti-Coronary Club experiment were published in 1966 in the Journal of the American Medical Association.2 Those on the Prudent Diet of corn oil, margarine, fish, chicken and cold cereal had an average serum cholesterol of 220, compared to 250 in the meat-and-potatoes control group. However, the study authors were obliged to note that there were eight deaths from heart disease among Dr. Jolliffe’s Prudent Diet group, and none among those who ate meat three times a day. Dr. Jolliffe was dead by this time. He succumbed in 1961 from a vascular thrombosis, although the obituaries listed the cause of death as complications from diabetes.
3) Legal Issues in Disaster Medical Standards. Mass Casualty Incident Management (MCIM) Conference in New Orleans, July 2011, Edward P. Richards, III, J.D., M.P.H.