Science and Running: How Do You Know What’s Good and What’s Junk?

img_0348In 2010, thanks in large part to Christopher’s McDougall‘s Born to Run, barefoot running became the newest running trend. Everywhere I went I heard about the health benefits of running barefoot. I heard it all, from “You’ll never get injured again!” to “You’ll be so fast!”. And research seemed to back up the barefoot running phenomena, yet I remained stubbornly unwilling to try. I was a bit relieved, then, when other research emerged that contradicted the earlier study’s findings, but not before Vibram sold millions of its five-finger minimal running shoes. These later findings resulted in a class-action lawsuit against the company.

This is just one example of companies and other interest groups using purported research to promote an agenda and get into our pocketbooks. We need to examine all the claims we see every day in running magazines, on blogs, and especially those made by marketers that say that we runners must be doing this or that thing or use this or that product if we want to be a healthy high-performing runner.

Why do we need to give a critical eye to research? Media reports on science to gain eyeballs, so it’s not always in their interest to report good science (think: clickbait). Marketers report on, spin, or even manufacture science (I’m looking at you, Gatorade) to sell their products. Think about all the things that media and marketers have used science to claim we need to stay healthy and perform our best:

I’m not saying either way that any of these things are good or bad. In fact, I swear by a few of them as an experiment of one (foam rolling and sports massages), but not all of them (orthotics and compression socks). The research and subsequent articles espousing the benefits of these products is often contradicted in one or more studies. Naturally, the company that makes those products doesn’t show you that research.

For example, while we see many of our favorite athletes rocking knee high compression socks, the research is mixed about the benefits of these garments. One study failed to find significant differences in performance between sessions with and without compression. Another group did not find differences between groups during a fatiguing single-leg hopping task. Researchers have had a little more luck with cyclists, citing limited, but likely physiological benefits.

Additionally, most of us runners have been inundated with comments from well-meaning friends citing “research” that running is bad for us, something that we sense not to be true. Running isn’t shaking my uterus loose, my heart is quite healthy, and my knees are in fantastic shape. My point? We need to be taking the so-called running research with a grain of salt.

Here’s how to investigate the validity of scientific claims about running products and practices.

Find the studies

Sometimes there is no study directly on point. For instance, when we were researching the claims some cryotherapy proponents make about how it can boost metabolism and contribute to weight loss, we could not find studies about cryotherapy and weight loss. However, we did find studies on the benefits of cold therapy in recovery from hard workouts that mention metabolic effects. Were these the studies that the marketers were using to make their claims? Probably. One way to find the research behind the claims is a simple google search. Even with my access to university databases, I always go to google first to see what is being said about the topic. Most of the time, googling the topic, say using the keywords “cryotherapy” and “athletic performance,” will pull up the different articles written based on the research. From there, you can see the range of articles not immediately available in an article promoting a single product or message. However, if your google search isn’t pulling up any research, you might have to tap into your friend in college or who works in higher education. They will likely have access to the databases where the journals publishing this research can be found and can guide you towards the research that will answer your questions.

Ask questions

Once you find a study that might say what the articles or marketers claim they say, then you must analyze that study by answering some questions like:

Is correlation being stated as causation?

Carefully review whether the research is being used to state causation when, at best, the research only shows that two factors are related (correlation). For example, a study could find that 90% of runners eat ice cream and claim that the act of eating ice cream leads to running, which would be awesome, but is not, in fact, what the research found.  Similarly, much of the research that relates running to knee injuries are correlational (and recently contradicted). Those studies found that these injuries occurred frequently in runners, but they were unable to state that running (and not other factors, like prior injury and genetics) caused knee problems.img_0349

Who is included in the sample?

By sample, what I mean is who participated in the study? For example, if the researchers want to know how a certain supplement affects recovery time but they only talk to runners in their mid-20s, then the research only really applies to runners in their mid-20s, not to runners in their mid-50s or teens.

How many people actually participated in the study?

If the researchers studied thousands of people, you probably won’t be as skeptical as you would be if the study only included 30 people. Based on the sample size, were enough people studied for you to feel it necessary to change your behavior? Most of the time, if relatively few people were studied, I’d say probably not.

Who funded the study?

Although valid research may be funded by a corporation, research funded by a company with a stake in the outcome should always warrant a deeper look and a healthy dose of skepticism.

What do the findings actually say?

A famous psychology researcher once said that research should make sense. What he meant is that most research isn’t shocking or life changing, so we should be able to read the research and think yeah, I already knew that. Use your logic. Do the findings make sense with your own experiences? If not, then take a deeper look and search for contradictory evidence before making any changes to your routine.

Is the research stated as incontrovertible fact?

Good research, no matter how large the study is and truly random the sample is, will never state that something is fact. It will say things like, “the data supports the alternate hypothesis that finds that …” or, “research suggests that _____ happens”, but never things like “x causes y.” Why? Because no matter how many people they talk to, unless they talk to every single person in a population (e.g. runners), they can’t know for a fact that something causes or predicts something else. It’s not good science to state a finding with absolute certainty.

What does the other research say?

Good research will include a list of other studies that have similar and contradictory findings. For similar research, it’ll summarize how the new findings fit into that research. And for the contradictory research, it’ll explain how the current study is different – and will often suggest that future research examines those contradictions. Because, as I stated above, good research is never fact. It will always state, usually explicitly, that more and better research should be done.

***

For the most part, good science won’t conclusively defy logic, it will reinforce what you already intuitively know to be true. If it doesn’t (and doesn’t fit into your lifestyle), keep on doing what you’ve been doing. But if it makes sense and maybe addresses a problem or concern you have, do a little legwork before you put your money where your mouth is.

What running research have you seen over the years that you questioned? How do you decide what research to believe?

*Thanks to Pesto, our PhD to be, who contributed to and reviewed this post for Type I and Type II errors. 

Ultrarunner, yoga teacher, academic, and feminist. I write about ultrarunning, feminism, and the intersection of running and life.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

6 comments

  1. When it comes to anything related to running, I usually just bug Salty herself, or check the archives here 🙂 I’m a librarian, so for everything else, I usually research too much. (and the questions you suggest to ask are spot on 🙂

  2. YES!! As a scientist, I am constantly trying to (gently) remind my family and friends to look at things with a critical eye; it takes a lot of work to read the original articles, but it is worth it! I believe Pubmed.org, the government’s open access database to peer-reviewed journal articles funded by the NIH (National Institutes of Health) is available to everyone; even reading the abstracts to articles not available outside universities can be helpful, although not entirely thorough. I loved John Oliver’s take on interpreting scientific studies:
    https://www.youtube.com/watch?v=0Rnq1NpHdmw
    Long, but so, so good. Thanks for bringing a little fact checking to our running science!

  3. I’m definitely the nerd that does lots of research on things, but also take some things based on my own experience. Example being compression- there is such mixed results in studies. While I don’t believe everything doctors say, I do believe that compression socks at hospitals etc have helped reduce blood clots etc. That doesn’t affect me as much obviously, but it does weigh on my mind. I love, and use compression because I personally feel it helps me. At some point I have to say, it makes me feel good so why not?

    I love that you pointed out asking who is funding the research. I think that is incredibly important to note, is this someone who stands to make a lot of money based on the outcome (or lose money). One of those random quotes that I really like is “don’t base your decisions on the advice of those who don’t have to deal with the outcome”- I think this definitely applies. Some companies don’t care if it helps you or doesn’t help you as long as they aren’t losing money, and even better- they are making the money.

  4. This is great! Thank you for this public service 🙂 I’m like Barley, I like to do my own research, but ultimately go by my own feeling about what works for me (I’m also a fan of compression socks because they feel good, don’t care if they make me run faster or not!)

  5. Thank you for this- there’s so so so much research going around on every topic imaginable that it’s really hard to figure out what’s real and what’s bull! I do find a caveat to this is to look at what works for you. For instance, I don’t find ice baths to help my recovery time when I have sore, tight muscles. But I do find super hot baths to help. But when I’m actually injured, it’s ice all the way. My physiotherapist agrees with this, and tells me to use myself as an experiment of one!