A no-difference conclusion arises after positive findings are explained away

The recent articles proclaiming the death of organic foods could as well be written by big agribusinesses. The  reporting raises a whole bunch of questions. Here's the NYT on the study (link) and here's Time (link).

Here are some points to ponder:

1. There is no new research. Time's headline gets this wrong even though in the article itself, the author says the research is a review of existing studies (so-called meta-analysis) but on this point, the NYT is clear, the study is an "extensive examination of four decades of research".

2. Despite the headline claim that organic food is no more nutritious and no less contaminated than non-organic food, both articles devote the majority of words to reporting the statistics that contradict the claim. We learn that organic food has less pesticide residue, less bacteria, more omega 3 fatty acids, more phosphorous, more phenols, etc.

3. The researchers then tell us why none of those negative findings can be trusted. This runs the gamut from the difference is not practically significant because the levels are too low to matter, or because the levels are so high it doesn't matter, or in the case of bacteria, because bacteria will be killed during cooking.

4. A particularly shocking claim is the one cited by NYT to dismiss the more phenols finding: 'While the difference was statistically significant, the size of the difference varied widely from study to study, and the data was based on the testing of small numbers of samples. “I interpret that result with caution,” Dr. Bravata said.' Statistical significance was invented for one purpose, which is to deal with sampling variability, and sample size. It makes little sense to blame those two factors after testing for significance, otherwise they shouldn't bother testing for significance. In addition, since this is a meta-analysis, they could easily have thrown out studies in which the sample size is considered too small, or the sampling variability is too high -- doing so upfront will reduce the suspicion of cherrypicking.

5. In fact, these researchers seem to already have hit the tip of an iceberg. At the end of the NYT article, they admitted that they erroneously omitted a study that showed organic food had a benefit. Given that the selection of which studies to include in a meta-analysis is such a crucial step, it does not help their credibility to have committed this error.

6. As far as I can tell, none (or almost none) of these studies are randomized controlled trials (RCT) in which the health benefits of organic foods and non-organic foods are assessed. This is quite surprising given the assertion that the two groups of foods are the same. Given that it is likely that no one would die from such a study, there just isn't any reason why we should rely on observational studies. In fact, according to Time, only 17 of the over 200 studies included in the meta-analysis measured "health outcomes"; all the other studies compared "nutritional content".

7. The bulk of both articles accepts the underlying logic that consumers buy organic food because of better nutrition or lower contamination. I know many people who buy organic food to support small businesses and others would prefer to know that the chicken they are eating were able to run around rather than get cooped up inhumanely. This is a fundamental flaw of a study that is being interpreted as a reason not to buy organic food. The NYT reader would have to read all the way to the end of the article to see this point raised.

***

For those nutritional scientists out there, do you know why RCTs aren't popular in your field? Almost everything we read from the newspapers report from longitudinal observational studies that really do not seem robust enough, resulting in all kinds of contradictionary conclusions, and embarrassing reversals.