This Spring the Institute of Diet and Health, in Germany, reported that their researchers had come up with an amazing way to lose weight. Simply add 1.5 ounces of dark chocolate to to a low-carb diet, and you’ll lose more weight.

The study’s lead author, Johannes Bohannon, Ph.D., shared the exciting news. Their research had found that participants in a study examining the health effects of a low-carb diet lost weight 10% faster than the control group when – yahoo! – they also ate chocolate. Reporters all over the world ate up the story, shared the news in big, splashy (chocolatey) headlines. You may even have seen it on your Facebook news feed: Slimming down by eating chocolate is a dream come true.

Except it’s not true.

Here are headlines from just some of the news outlets that picked up the story and ran with it. Notice that even the “health” magazines didn’t do right by their readers.

Johannes Bohannon is actually John Bohannon, a science journalist who does have a PhD but not in nutrition. He and some colleagues wanted to demonstrate how easy it is to turn bad science into the big headlines behind diet fads. They did recruit participants (16), and randomly assigned them to one of three groups: one that ate whatever they normally would have, a low-carb diet, and a low-carb with chocolate diet. The participants did spend 21 days on their assigned diet (one dropped out). Then the “researchers,” including a doctor and a statistician/financial analyst friend in on the prank, took measurements – things like daily weight, cholesterol, sodium, blood protein levels, sleep quality, well-being, etc. – from each of the participants. Data from those 18 measurements was crunched. With that many things to compare, they lucked upon a statistically significant  difference: Participants in the two groups on the low-carb diet lost about 5 lbs apiece, with the chocolate eaters losing 10% FASTER. Chocolate eaters also had better cholesterol readings and higher well-being survey scores.

Great news, right? John explains the problems with the study in an article he wrote for io9.c0m:

Here’s a dirty little science secret: If you measure a large number of things about a small number of people, you are almost guaranteed to get a “statistically significant” result.  Think of the measurements as lottery tickets. Each one has a small chance of paying off in the form of a “significant” result that we can spin a story around and sell to the media. The more tickets you buy, the more likely you are to win. We didn’t know exactly what would pan out—the headline could have been that chocolate improves sleep or lowers blood pressure—but we knew our chances of getting at least one “statistically significant” result were pretty good.

It’s called p-hacking—fiddling with your experimental design and data to push p under 0.05—and it’s a big problem. Most scientists are honest and do it unconsciously. They get negative results, convince themselves they goofed, and repeat the experiment until it “works.” Or they drop “outlier” data points.

But even if we had been careful to avoid p-hacking, our study was doomed by the tiny number of subjects, which amplifies the effects of uncontrolled factors. Just to take one example: A woman’s weight can fluctuate as much as 5 pounds over the course of her menstrual cycle, far greater than the weight difference between our chocolate and low-carb groups. Which is why you need to use a large number of people, and balance age and gender across treatment groups. (We didn’t bother.)

You might as well read tea leaves as try to interpret our results. Chocolate may be a weight loss accelerator, or it could be the opposite. You can’t even trust the weight loss that our non-chocolate low-carb group experienced versus control. Who knows what the handful of people in the control group were eating? We didn’t even ask them.

Whenever there are rules, there are people set up to break those rules and it’s even true in science publishing. John “Johannes” Bohannen submitted the resulting “scientific” paper to a list of fake journal publishers, and 2 weeks after paying “The International Archives of Medicine” 600 euros (about $700), the article was published.

John thought that his fellow science journalists would catch on to the prank quickly. There is no Institute of Diet and Health, just a website, and googling Johannes Bohannon would reveal he didn’t exist either. Plus, if a journalist checked the article with any scientist, they would discover the silliness of the study. But he was wrong.  It turned out that just about everyone swallowed this one hook, line, and sinker. Some even just copied and pasted the press release he sent out about the published article.

So what is the point? Folks don’t ask enough questions when they hear what they want to hear. This isn’t just true for nutrition science, in which practitioners say it is especially difficult to get meaningful outcomes. We all need to ask questions, especially when something sounds too good to be true. And that’s where John found hope. Many of the readers commenting on the chocolate articles were skeptical and asked the questions the reporters should have asked.

We try to ask all the right questions before we share science with you. But if something doesn’t look right to you, we hope you’ll ask us more questions. Together we can get to the bottom of it all. Which, of course can be done over a piece or two of chocolate.