Am I shittin' you? Learn to be a skeptic!

Learn to be a skeptic!

Why you cannot believe what you read about medical studies.

In my last blog post I promised to tell you why you shouldn't trust any study results, particularly when you didn't read the study yourself. It has to do with the methods of biomedical research. To make my point, I'll take the gold standard research method, the double blinded randomized controlled trial, or RCT. 

Let's say we want to test a drug, which is supposed to lower blood pressure in those who suffer from hypertension. The researchers have decided to enroll, say, 100 "subjects". That's what we typically call the people who are kind enough to play guinea pig in our studies.   

The researchers will first do a randomization of subjects into one of two groups (very often it is more than one group, but to keep it simple we will assume just two groups). What we mean with randomization is that we randomly assign each subject to one of the two groups. One group - the intervention group - will receive the drug, the other group - the control group - won't. What they get instead is a sugar pill, a placebo. 

With the randomization we want to make sure that, at the start, or baseline, both groups are indistinguishable from each other with respect to their average vital parameters. For example, if we were to calculate the mean age, blood pressure and any other variable for each group, these mean values would be not different between groups. That's important, because we want to isolate the effect of the drug. We don't want to worry at the end whether the effect, or lack thereof, was maybe due to some significant difference between the groups at baseline. 

Once the randomization is done, we organize the trial in such a way that neither the "subjects" nor their physicians and nurses know whether they get the placebo or the active drug. Both sides are blind to what they get and give, which is why this set-up is called double-blinded. That's an important feature, because a researcher often goes into a study with a certain expectation of its outcome. Either that outcome supports his hypothesis, or it doesn't. To eliminate the risk of, more or less subconsciously, influencing the study towards a desired outcome, double-blinding is very effective tool.

Fast forward to the end of our trial. We have now all the data in hand to compare the two groups. After unblinding, the researchers will compare the two groups with each other. In our example, they will compare the average, or mean, of the blood pressure values of all the individuals for each group. If the intervention group's mean value is lower than that of the control group, then it is plausible to reject the null-hypothesis, that is to REJECT the idea that the drug is NOT as ineffective as the placebo (we are, of course, assuming here that the sugar pill didn't lower the blood pressure of the control group). 

There are statistical tools to determine whether the difference between the groups may just be a chance event, or whether chance is a very unlikely explanation. We can never rule out chance completely. Now, when we are confident that it is the drug and not pure chance, which has lowered the mean blood pressure in the intervention group, we write our paper to present it in one of the medical journals. 

If the subject is a little more sexy, than just lowering blood pressure, there will sure be some journalists who pick it up and report to their readers that, say, eating chocolate makes you slimmer. I'm not kidding. This headline very conveniently went through the media shortly before Easter this year [1]. Good for Hershey who are running it of course on their webpage. And in the media it reads like it did in the Irish Times: "Good news for chocoholics this Easter. Medical Matters: No need for guilt over all those Easter eggs."    

I'm not going to comment on the media geniuses, because it's their job to put an angle on every story, so that YOU find it interesting and read their stuff. But since I'm sure you'll follow these links, just let me warn you: the chocolate study was an observational study, not an RCT. And one thing we MUST NOT do with the results of observational studies is to confuse association with causation. Only when we conduct an RCT, where the intervention group eats chocolate and the control group doesn't, might we be able to determine whether there is a causal link. And for obvious reasons we can't blind the subjects, to whether they eat chocolate or not. But I'm digressing.

Back to our blood pressure study. When we compare the group averages, everything looks very convincing. And sure enough, as researchers we are happy with the results, and we are perfectly correct, when we conclude, that this medicine does its job. 

But will it do it for you? When you are hypertensive? You might be wrong if you say "Yes". And you will be wrong more often than we, as researchers, or your doctors care to admit. For one simple reason: The variability of effect within the group. You give 50 people the same drug, and I bet with you, and I'm not the betting type, that you'll have 50 different results. 

The mean value of the entire group glosses over these inter-individual differences. Let me give you an example from a study performed on 35 overweight men, who were studied in a supervised and carefully calculated 12-weeks exercise program, with the intention of reducing body weight. The mean weight loss was 3.7 kg. That was almost exactly the amount of weight loss which the researchers had expected from the additional energy expenditure of the exercise program. But when they looked at each individual, it became clear that the group mean doesn't tell you anything about how YOU would fare in that program. 

First of all, the standard deviation was 3.6 kg. Now, a standard deviation of 3.6 kg simply tells you that approximately two thirds of the participants experienced a weight loss anywhere between 3,7 kg (the mean) minus 3.6 kg and 3.7kg + 3.6 kg, that is between 0.1 kg and 7.3 kg! That's a lot of kilos. And what about the remaining one third of those participants? They are even further from these values. In this case the greatest loser went down by 14 kg, and the biggest "winner" gained almost 2 kg. A spread of 16 kilos!

Here is the graph which shows you the change on body weight and fat for each individual participant. Which one would you be?

This effect is what you do not see when you don't read the studies. And in most studies, it isn't made obvious either. 

Which is why, you shouldn't be surprised to learn that most major drugs are effective only in 25-60% of their users [2]. The same goes for weight loss drugs and interventions, for almost everything we study in biomedicine. 

That's not a problem for us in public health. Because a drug, which works in 60% of the patients, helps us reduce the burden of disease in our population. Public health is not interested whether you are one of the 60% or not. But you are. And that's why I believe not only medicine, but also prevention must be individualized.

 Which is why the GPS to chronic health, which I currently develop, is all about helping you find your individual path to your health objectives.

Why not have a look at it, and maybe even try it out? 

References

1.           Golomb, B.A., S. Koperski, and H.L. White, Association Between More Frequent Chocolate Consumption and Lower Body Mass Index. Archives of Internal Medicine, 2012. 172(6): p. 519-521.

2.           Wilkinson, G.R., Drug Metabolism and Variability among Patients in Drug Response. N Engl J Med, 2005. 352(21): p. 2211-2221.