In November of 2012, the journal “Food and Chemical Toxicology” published a paper by French scientist Gilles-Eric Séralini and his colleagues called “Long term toxicity of a Roundup herbicide and a Roundup-tolerant genetically modified maize.” The study was developed because Séralini and his team were interested in indications of toxicity found in the raw data of an earlier 90-day study done by Monsanto. The results of Séralini’s two-year study showed as much as 5 times more liver and kidney disease, and 2 to 3 times more tumors in rats that had eaten Genetically Modified (GM) corn and that had drunk water containing Roundup.
In August of 2013, the journal retracted the article after reviewing the study design and the data it was based upon. Among the flaws they noted was that there were too few animals in the study, and the type of rat used, the Sprague Dawley, is prone to cancers and tumors. Folks on both sides of the issue have cried foul, some for the publication of the article in the first place, and others for its retraction. Both sides continue to present data and arguments to support what they believe to be the truth. And that’s where this gets interesting, pointing out the importance of understanding the scientific process and how difficult it can be to get everything just right. It also brings up the specter of profit and politics, two issues that are hard to remove from the daily lives of humans, leaving us with the question: Who should we believe?
We’ll try to help you with that question by using this controversy as an example of how to dig through the data and the claims to get a little closer to that elusive creature: THE ABSOLUTE TRUTH.
How To Read A Scientific Paper
Scientific papers are not meant to be light, entertaining reading. They follow a formula that makes sure you know precisely what steps were taken as the study was carried out, how the data was gathered, and what kind of statistical analysis was done on the data to determine how powerful the results are. It’s the scientist equivalent of Hansel and Gretal leaving breadcrumbs: If everything was done correctly, we should be able to do the exact same steps again and get pretty much the same results. By understanding exactly what was done, it also makes it possible for researchers to build on the results of a study by doing something slightly different, or for a longer timeframe, or with more animals. Well-done research will also be completely transparent about the funding of the research and the affiliations of the authors so that we can view the work while keeping in mind any biases that the authors may have had. It’s a system of checks and balances.
Most studies answer one small question, but raise many more questions. Thus most research papers end by saying something like “More research is needed.” This isn’t simply a ploy to get more money for more research. It’s actually a sign of humility and an acceptance that one paper is not going to give us the answer to life the universe and everything.
Here are the parts of a good scientific paper:
- Title
- Authors and their affiliations: This is generally everyone who participated in making the study possible and writing up results. Because we’re known by the company we keep, this gives you a first look at any kind of bias that might be included in the paper.
- Abstract: This is a summary describing what the study was and its findings.
- Introduction: Here’s the background leading up to the study. It often cites previous studies so you know what the authors were basing their work on and what they considered as they developed their experimental design. By showing us this, they’re allowing us once again to screen for bias.
- Materials and Methods: The devil is in the details and this is where the devil resides in a research paper. If you want to do the test again, just follow the instructions you find here. Though it can be dull reading, it’s incredibly important to determining the validity of the data.
- Results: This is the data and the statistical analysis of it.
- Discussion: Here’s where the authors tell you what they derived from their study.
- References: This is like the genealogy of the paper. It tells you what other papers the authors used to inform their process, study design and discussion. They are noted in parentheses throughout the paper, and then the full citation is shown so that readers can check out the papers on their own.
All in all, the scientific process and the papers that come out of the studies are like building a step-by-step map that shows what a group of researchers did and what they learned, and it allows anyone to look at all the information to see if their results and conclusions were valid. Papers are reviewed by scientific peers and if they think that everything was done properly, the paper is published in a journal. Publication is like a certification that says, “Yes, you can use this information to inform other projects or changes in management.”
It is not unusual for papers to be controversial even after publication. Discussions, like the one that is ongoing about Séralini’s paper, are good because they ensure that no one is making things up and that we continue to work through problems until we have something truly substantial on which to base our decisions. These discussions can unveil additional information and biases. That’s why THE ABSOLUTE TRUTH is so elusive. Everyone has a bias, and there’s always more to know.
Let’s see how this all works in the Seralini paper controversy.
What About That Rat?
Let’s start with the type of rat used for this study. There are about 200 strains of rats created through breeding so that they have traits in common. This reduces variability among individuals in a study design and allows researchers to be more precise. For example, the Zucker rat is a strain developed to study obesity and high blood pressure. Then there’s the Lewis rat strain, commonly used for transplantation and arthritis/inflammation research. For research on macular degeneration, researchers prefer the RCS rat which is the first known animal with inherited retinal degeneration.
The Sprague Dawley rat, which was used for this study, is an albino rat. They live about 2 to 3 years, with females typically surviving longer than males. Research done on this rat strain indicates that at about 1.5 to 2 years of age 87% of them will have developed some sort of tumor. These tumors are found in pituitary and adrenal glands and in the case of female rats, in the mammary glands.
The Sprague Dawley is the most widely used rat in biomedical research, particularly for toxicology research when we’re trying to understand the effects of chemicals on living creatures. The National Toxicity Program in the U.S. uses the same rat, from the same source as the Séralini study, for long-term toxicity and carcinogenic studies. The journal that retracted Séralini’s article had previously published a 90-day study by Monsanto using the same rat and showing no negative effects of eating GM corn. Monsanto’s study was also used to inform the European Union’s decision to authorize the use of glyphosate (Roundup) in member countries.
So, if everybody uses the Sprague Dawley rat for this kind of research, how do we know that changes in the number of tumors reported by Séralini are due to the chemicals they’re eating, or simply a result of their tendency to get tumors as they get older?
Controls
You probably remember this from your science classes in school. If you want to know if something you’re doing is making a difference, you need to have a “control” to which nothing is done. Then you can compare the results of a change to doing nothing. As a good scientist should do, Séralini included a control group in his study. When the study was finished, he could compare the number of tumors between the female rats in the “treatment” groups and the female rats in the control group. His study showed that by the beginning of the 24th month, and depending on which level of chemicals the animals had been fed, 50 to 80% of the rats had tumors. Meanwhile, only 30% of the control group had tumors. His conclusion was that a treated rat was 2 times more likely to have a mammary tumor.
Ah! But any time you come to a conclusion, it’s based on statistics, or as many people like to say: “Lies, Damn Lies, and Statistics.” If you’re going to do a study, you have to be sure that the size of your treatment and control will give you “statistically valid” results. Critics of Séralini’s study say that he used too few rats and thus his results are useless. Séralini pushes back by saying “I used the same number of rats as Monsanto used in their study,” adding that in fact Monsanto did have 20 rats in each group, as compared to the 10 he had in each group, but that Monsanto only reported on the results of 10 members in each group. No one knows how Monsanto chose which 10 rats to use and which to throw out in each group, leaving us with interesting questions about that study’s validity.
Hmmmm……it seems like there’s more to look into here, and that’s exactly what we’ll do next week. It might seem that statistics is a boring topic, but in this case the stakes are high. How well scientists are doing their jobs will tell us if we’re safe eating all the foods that come with GM ingredients. People’s lives and livelihoods are on the line, and that’s why it’s so important for us to be able to drill down into the data, and the politics. We may not come up with THE ABSOLUTE TRUTH, but by understanding the process, and the ongoing discussion, we can be part of the community of science referees that keeps everyone honest.
Stay tuned for our next, exciting installment: Lies, Damn Lies, and Statistics: GM Corn, Roundup, and Tumors
Thanks for getting this conversation started here and we look forward to your next installment. I’d like to guess that the “scientific protocol” that allowed Monsanto to pick which ten rats it wanted to report on was originally created by researchers who did not want to report the real truth of their findings. Sure, “they” will rattle off 25 reasons this “protocol” is OK and scientific but they are just trying to fool themselves as well as the rest of us. Just another example of how broken the system is. We will continue growing quality forage and avoid using herbicides and pesticides and share our methods with each other. Hopefully more and more people will do the same. Thanks.
I didn’t do a good job when I wrote the sentences about Monsanto testing only 10 rats out of its group of 20 and I left the impression that they may have done something bad. I have since checked out the protocol (OECD 452) more fully. The protocol allowing researchers to run research that includes 20 animals in a group but to only have to do blood work on 10 of them was developed by the Organization of Economic Cooperation and Development. The purpose of this protocol, like all of the others, is to make sure that scientists follow proper procedures. Observations are done on all animals in the group, but blood work is only done on at least 10 males and 10 females from each group. The same rats are tested throughout the life of the study so researchers can’t just pick animals to get the results they want. Blood work is expensive, so doing it on only some of the animals saves money while still providing a study with statistical validity. All the animals are sacrificed at the end of the study and necropsied and information is gathered on all of them at that point.
It’s unfortunate that this kind of controversy tends to reduce trust in scientists. Of course each scientist starts with a hypothesis, and maybe even has a preferred outcome. But my experience working with researchers has demonstrated that overall they are trustworthy, and the system they work within has enough checks and balances that those who cut corners are found out sooner rather than later.
Thanks Kathy, for the clarification on the protocol and how it is carried out. You are fairly bringing out issues in a easy way to understand.
Chip
The one thing missing in this debate is the confirmed fact that weeds are becoming resistant to glyphosate which was a given at the beginning which Monsanto knew. Chemicals are always going to be defeated by the natural world. And the geneticists at Monsanto ignored this.
Ron,
I do not believe Kathy is riding into the valley of death. She is explaining research methods and that I appreciate. The part I am glad she brought out was Dr. Huber used ten rats and Monsanto used 20 and cherry picked the 10 to use for the results. That is, in my thinking, unethical. I can make thinks look much better be picking results that fit what I want instead of going with what is found.
Hi Kathy,
Are you sure you want to open this can of worms on science? I’m sure you realize that this subject (science) is fraught with political and economic overtones that preclude changes being made about the “truthiness” of science. I’ve included a few links to articles that show that much of what is viewed as the truth (something that science is tasked with ferreting out) is nothing more than what a person has been lead to believe. I rarely believe anything that I read/hear/see in the mainstream media anymore – especially if there is any economic ramifications to the subject at all. Instead, I will research on my own to determine – as much as possible – what the facts are on the subject. I doubt most people have either the time nor the inclination to do so themselves; too much “backfire effect”!
I think you’re riding into the “Valley of Death” on this subject.
http://www.spaulforrest.com/2011/06/why-do-people-believe-stupid-stuff-even.html
http://www.merchantsofdoubt.org/
http://dementedagitprop.com/
http://www.ucsusa.org/assets/documents/scientific_integrity/how-corporations-corrupt-science.pdf
http://www.nytimes.com/2011/12/23/opinion/krugman-the-post-truth-campaign.html?_r=0
http://mycommonsensepolitics.net/index.php?option=com_content&view=article&id=3154:the-science-of-fox-news-why-its-viewers-are-the-most-misinformed&catid=19:the-really-bad-mean-folks&Itemid=30
Well, having opened this can of worms a bit, and on a study that is very controversial, I agree that it’s very scary. In fact, I’m working hard on both understanding and explaining how statistics worked in this study and should work in general for the follow up article.
The fact is that every single human being comes with bias of some sort. The purpose of the way we go about gathering information is “supposed” to give us the best shot possible at being as transparent, authentic and as truthful as possible. What I hope to do is help us all be more careful consumers of information, and to notice when biases are showing, whether it’s our own or someone else’s. I’d also like to promote more thoughtful dialogue, a little trust, some self-awareness, and a lot less name calling and insults. I believe that’s the path to better solutions to all the issues we face.
Good article. I hate the fact that a new term “good science” is now being used and my definition of “good science” is a particular study or research providing information that agrees with someone or some groups point of view.
Too often now, when we hear that “a scientific study concludes:” we just groan and discount the information.
Interesting, thank you. One aspect which I felt deserved more attention was the animal welfare issues with this paper. I blogged with my thoughts http://gmfromthefence.wordpress.com/2013/12/02/retraction-of-seralinis-controversial-study-on-gm-toxicology/
Kathy,
When Monsanto used 20 rats and picked the ten to use for their results that would seem to be not only unethical but manipulating the results. If someone can cherry pick the results that is not true research in my thinking.
I have always assumed there was a standard protocol to follow. Is the Monsanto research method allowable in the scientific community?
This was a great article. Thanks for delving into the two research programs to bring out the differing methods. All the fighting going back and forth in the news is hard to understand without clarification.
Hi Chip!
I’ve been doing a lot of reading on this topic and have learned that there is a scientific protocol that allows for the testing of only 10 of the rats, making what Monsanto did perfectly fine. I’m not sure why, because I’m not a statistician which is what makes trying to write about this for the regular person, like me and you, so very hard.
I’m glad to see the issues discussed, but would point out the following- 1)there are several statistical strategies for dealing with experimental evidence. Likewise numerous statisticians -and of course THEY do not speak with unanimity. 2)Following one “recognized” protocol carefully without deliberate cheating is important, if only so that others can evaluate your work. But suppose the chosen protocol was not entirely appropriate, and a different one gives a different answer? In any case, independent replication will be needed to sort these problems out.
So what is a concerned individual to do?
Well he/she can look at the evidence to the extent time permits, and make a TENTATIVE conclusion, subject to modification as new info- emerges. It saves time to defer to opinions of some experts ,whom one believes are better informed on the issue. Of course one’s opinion of any expert, or group thereof, can also change, especially as circumstances change and new info- emerges.
‘Twas ever thus -We all live with our beliefs and the knowledge that others disagree. These beliefs may be better informed by statistical analysis or indeed refuted (at least for the time being.) Daniel Kahnemann has a best seller– THINK FAST, THINK SLOW, that guides one thru to some realization of how often we deceive ourselves with or without statistical help.