« Previous article
Newer article »

The problem with retrospective studies: Why what’s good for you changes

2008 August 23

By David Annis
Article ID: 1245

 
Download MP3  Download audio: Play Now

I often hear the complaint how scientists “change their minds” about what makes a healthy diet.  Why does that happen? 

When studying human health, especially studies involving diet over an extended period of time, it’s difficult if not impossible to get subjects to agree to allow researchers to control what they eat and drink, so cohort studies are done. (A cohort study is a study of a group of people who share common characteristics over a period of time.) Furthermore, since retrospective studies are cheaper and quicker than prospective studies, studies are often retrospective.

Studies done this way, however, have several problems.  The largest issue may be that correlation does not imply causation.  For example, a study at the Utah Health Sciences center found that the more diet soda in a person’s diet, the more likely that person was to become overweight or obese.  “After adjusting for age, sex and ethnicity, Williams found that regular soft drinks were no longer significantly linked to the incidence of becoming overweight or obese, but diet soft drinks were.”

The media will pick up a press release about this study and report that diet soda can make you fat, but correlation does not imply causation.  For example, soft drink sales may be highly correlated with shark attacks, not because soft drinks cause sharks to attack, but because hot weather makes people buy soda and go swimming.

Let’s think about what may be going on in this case.  We understand that weight gain is caused by consuming more calories than you expend, and since diet soda has no calories, while it is correlated with weight gain, it does not cause the weight gain.  However, I would guess that soft drink consumption is highly correlated with consumption of fast food, which is highly caloric.  Of the people that consume a lot of fast food, those that are starting to experience weight gain are more likely to choose diet soda than those that haven’t experienced weight gain.  So, diet soda consumption is probably highly correlated with people who have a slowing metabolism and a habit of eating fast food.  Perhaps, it is not the diet soda that is the problem, but the type of restaurant where large portions of diet soda often accompany the food.

The next big issue with cohort studies is that they often look at hundreds or thousands of factors since it is not very efficient to run a large study, gather data, and analyze it to look at only one type of food.  Scientific studies, however, rely on statistical analysis to determine whether something is true.  Even if your estimated error is less than one percent, in a study examining thousands of items some will appear to have an effect even though they are just statistical aberrations.  These false positives are then reported, and when a new study fails to confirm them as true, the press reports a scientific “change of mind”.

Finally, when a cohort study is retrospective, the problem is compounded since retrospective studies often rely on memory.  If you developed lung cancer, asthma, emphysema or other respiratory problems, you may be biased when asked by a researcher about the number of cigarette packs you’ve smoked.

Be careful with retrospective studies. Look at the detail. You might find that what was reported may not be what really happened.


[del.icio.us] [Facebook] [Fark] [Google] [MySpace] [StumbleUpon] [Twitter] [Yahoo!] [Email] More »

Other articles related to this topic:

from → Healthcare, Pop Culture

1 Comment leave one →
2008 August 26
Rick Smathers permalink

It is interesting that you cite a correlation between diet soda and obesity. In a recent study performed on rats, feeding them diet foods caused them to consume more calories over the long term. The theory is that the diet foods confused their bodies ability to self-regulate intake. http://news.bbc.co.uk/2/hi/health/6933686.stm

So, while this is by no means confirmed and much more study is needed, it is possible that Diet Sodas DO contribute to obesity (and therefore the relationship may be causal). I think it is very premature to draw that conclusion though.

Obviously the value of finding correlations is that they point to anomalies that need further study to determine if causation exists. It is unfortunate that they are often misunderstood by those outside the field of study.

In my humble opinion, scientists and journals should take more responsibility for reporting their findings with a clear summary oriented for the press and the layman. They must also resist the tendency to exaggerate their findings with future funding in mind.

Similarly, the press needs to understand that until a study has been peer-reviewed and confirmed, any conclusions is draws are tentative at best.

Much of the general public can’t tell the difference between a single study pointing to an unconfirmed correlation and a well established theory with countless confirming experiments and studies. Even within a discipline, you run into a minority of “experts” who don’t completely accept the basic theories. Sometimes these rogue scientists can produce brilliant results (Ex: Einstein) and sometimes they seem to be suffering from denial (Ex: Behe).

Each scientific discipline (and even each scientist) must take responsibility for asserting what is established core theory from what is unproven conjecture at a level that the general public can understand.

Rick Smathers

Leave A Comment

Note: You can use basic XHTML in your comments. Your email address will never be published. If your comment contains more than two links, the comment will be held until an admin manually approves it.

Subscribe to this comment feed via RSS