Dr Sam Decombel is the chief operating officer of genetic research and development company MuscleGenes and has a PhD in Genetics from the University of Birmingham. She is based in Oxfordshire.
Once upon a time, as a science student – way longer ago than I care to admit – I had to trek down to the university library and hunt down dusty old journals to find and photocopy research papers with which to review a subject. With everything now online, this antiquated access to research is a distant memory. Even better, thanks to the advent of open access journals, where once we had to pay for this content, in many cases it is now freely available to us all.
As a result, when it comes to scientific claims we no longer have to take the word of the ‘expert’ on face value alone. We can go straight to the source material and make a judgment for ourselves.
But how should we approach this task, and what should we be looking for when assessing that rather complicated looking research paper they’ve referenced? Here are the five steps you should take to scrutinising research like a scientist.
1 Check for conflicts of interest
This will come as no surprise, but not everyone who conducts scientific research is impartial. Sometimes the researcher may be influenced by a secondary interest, such as financial gain. One common example is when the study is funded by a food business with a vested interest in the outcome of the research. Of course this doesn’t automatically make the research wrong (in many cases where the study has negative consequences for the funder, they simply don’t publish at all), but you do need to be more cautious in how you interpret the data.
All reputable journals require authors to disclose any potential conflicts of interest, which you should find stated within the article, usually at the back. Food for thought? Check out who the last-named author works for.
2 Consider the quality of the journal
Research articles or papers are published in scientific journals. Like all things, there are high-quality journals and not-so-high quality journals. One of the simplest ways to get a measure of how valued a journal is within its field is to look at its impact factor. This is a measure of how many times articles from that journal have been cited by others, and is considered to reflect the relative importance of that article, and hence the journal that published it.
Journal impact factor has its flaws. It can be manipulated by editors, and lesser-quality papers can slip through because of novelty or cronyism, and as a result it has been controversial. In addition, good research can be found in the lower impact papers, as it may have been deemed of limited interest at the time. Nevertheless, it provides a simple, quick indicator of quality, as papers published in high-impact journals are generally more rigorously reviewed, because of increased competition.
3 Don’t just read the abstract
As it suggests, the abstract is simply the author’s summary of the paper. It is intended to highlight their conclusions and the most interesting aspects of the study. As we’ve already established, not every author will be without conflict. So if it seems relevant, it’s time to dig deeper. Get this far and and you’ll already be doing better than 99% of those who claim to back up their theories with published research!
4 Go straight to the data
This is key. Consider how the study has been designed and whether it was appropriate for the research question being asked. Don’t let the author lead you through their interpretation before making your own, as it may prejudice the way you look at the data. Some key things to consider are: is it a human study, or an animal study? Animal studies provide excellent models for understanding basic physiological mechanisms, but it can’t automatically be assumed the results will also apply to people.
How many people was the study conducted on? In order to have confidence that the results are representative, it is important to have a large number of participants. Too few, and the results may just have occurred purely by chance. Studies on 100 people or fewer, particularly if the differences between the groups are small, should be treated with extreme caution.
Who was the study conducted on? Was it on men or women? A single ethnic group or multiple ethnic groups? Professional athletes or untrained individuals? Just because a hypothesis appears true (or false) for one group, doesn’t mean it will be the same for all.
What controls has the researcher employed? A scientific control is an experiment or observation designed to reduce the effects of variables other than the one that is under test. In other words, are they comparing their test group against something else, and is the comparison sensible?
Let’s consider the research question ‘does a high carb diet have a negative impact on health?’. An example of a well-controlled study in mice might be one in which we split inbred mice (so they have the same genetic background) into two groups. We would keep them under the exact same environmental conditions with the exception that one group was fed a diet that was 70% carbs, and one group was fed a diet that was 5% carbs.
The carb source would of course have to be controlled as well, to ensure any differences were not attributed to the food type or quality. As you can see, this methodology is obviously not so easy to apply to people, which is one of the reasons human studies can be so problematic.
5 Take your time to become an expert
It is important to think of the study in context to the bigger picture. Constantly read papers, and try to become an expert in the field by understanding such things as the basic biology of metabolism, or the statistical tests that are used to show significance. Conclusions cannot be made on a paper in isolation. In light of your analysis of the data and acquired knowledge, think whether the author has come to an appropriate and reasonable conclusion.
Do you think there may be other factors that might be affecting the outcome? Is it convincing based on the evidence presented, or more speculative and based on other assumptions? If the study has previously been repeated by another group with similar outcomes, this lends significant strength to a piece of research. Over time, you’ll start to get a feel for what constitutes a good paper, and which papers have clear issues. Taking the time to build up your expertise is something that will be of huge benefit to anyone who wants to excel in their chosen discipline.
6 Make up your own mind
We all know that many fitness writers, researchers and experts may not always be completely neutral when it comes to their area of interest, or even bother to read the original research to which they are referring. So next time you hear about the next best thing in fitness, don’t just take the author’s word for it – go to the source, get your science head on and make your own mind up.