Photo by Aman Deshmukh
Doing science is hard. The stuff we already understand isn’t science any more; it’s engineering, or maybe design. If you want to do research you’ve got to ask questions that we don’t already know the answers to.
The second-biggest problem with that is that you can’t be sure when you’ve got the right answer, because you can’t just check the answer key in the back of the book.
The biggest problem is that you don’t even know the right questions to ask, because your assumptions — the things you “know” that aren’t so — are getting in the way.
How to grow underwear mice
All the way through the late 1800s people believed in “spontaneous generation” – living things like rodents, maggots and fungus spontaneously appeared under the right conditions. For instance:
To create mice, a recipe called for dirty underwear and wheat grain to be mixed in a bucket and left open outside. In 21 days or less, you would have mice.
I am totally not making that up. (Click that link if you don’t believe me.) Real scientists put on their white coats and set out buckets of wheat and dirty skivvies and thought that the mice that showed up sprang forth from the grain.
All you had to do was sit and watch and you could see mice climbing into the buckets, or maybe put a net around the bucket to keep them out. Back in 1668 someone did just that, proving that maggots on steak came from flies and not from the steak itself. (Sorry if you were about to have lunch.)
But wait a second … I said people still believed in spontaneous generation until the late 1800s. What’s the deal?
The deal is that it was another 200 years before Louis Pasteur proved that the mold that grows in chicken broth also comes from outside. Scientists at that time couldn’t conceive of invisible creatures floating in the air that landed in the broth and grew.
It’s only obvious when someone else does it
Looking back, it’s easy to see the wrong assumptions they were making. The hard part isn’t disproving the assumption, it’s realizing that you’re even making it.
A good example of that in action today is the assumption that you can add up the calories someone is eating and know exactly what they should weigh. You see this in action every time someone says, “So if you cut 500 calories from your typical diet each day, you’d lose about 1 pound a week (500 calories x 7 days = 3,500 calories).” And that’s the Mayo Clinic saying that.
I happen to be pretty sure it’s not that simple, and there’s evidence to back me up. But the point I’m making here is what happens when you don’t question your assumptions.
People lie
People like to think they’re good and smart, so they’ll mis-remember what they’ve done so that it sounds better. Or they’re ashamed that they’re not so good and smart. Or they’re embarrassed to admit that they simply don’t remember. That’s why you shouldn’t trust observational studies that depend on food surveys.
But that doesn’t mean an unexpected result proves they were lying. Take a look at this passage from an editorial in the New England Journal of Medicine:
The reduction in caloric intake was also not sustained. Weight loss averaged 6 kg at 6 months, which fits reasonably with the planned daily deficit of 750 kcal. However, after 12 months, subjects started to regain weight, which suggests that they were eating more than planned.
In other words, researchers just know that if people stuck to the caloric target that they would have lost more weight. If they didn’t, it can’t be that the theory of calorie counting is wrong; it had to be that the subjects were lying about what they ate.
Assumption feedback
Take your first assumption and draw a conclusion, and now you’ve got even more faith in the conclusion because you already “proved” it. What’s that next step look like?
Participants who lost more weight attended more counseling sessions and adhered more closely to the prescribed dietary composition. These observations led Sacks et al. to conclude that behavioral factors rather than macronutrient composition are the main influences on weight loss.
But remember, they only assumed that the participants were lying. Now they’re using that as the basis for concluding that attending support meetings is more important for weight loss than what they ate.
Am I overstating it? Read more and you decide:
Weight-loss studies are behavioral studies; they require participants to eat less.
Right there in black and white: Weight loss can only happen by eating less. If you didn’t lose weight, you didn’t eat less. End of story.
Just as a thought experiment, imagine there are two possibilities:
- The only thing that affects weight loss is the number of calories eaten.
- Changing the balance of fats, carbs and protein can result in increase or decrease in weight even when total calories are held constant.
Now if that’s the question you’re trying to answer, could you possibly write this? (Emphasis added)
The inability of the volunteers to maintain their diets must give us pause. The study was led by seasoned investigators who were experienced in the performance of diet and drug trials. The participants were highly educated, enthusiastic, and carefully selected. They were offered 59 group and 13 individual training sessions over the course of 2 years. Nonetheless, their body-mass index (the weight in kilograms divided by the square of the height in meters) after 2 years averaged 31 to 32 and was moving up again. Thus, even these highly motivated, intelligent participants who were coached by expert professionals could not achieve the weight losses needed to reverse the obesity epidemic.
All these great conditions, experienced researchers, enthusiastic subjects, and they failed.
It couldn’t be that they were asking the wrong question. It was those stupid subjects who just couldn’t do what they were told and follow a simple diet.