Learn to be a skeptic!
Why you cannot believe what you read about medical studies.
In my last blog post I promised to tell you why you shouldn't
trust any study results, particularly when you didn't read the study yourself.
It has to do with the methods of biomedical research. To make my point, I'll
take the gold standard research method, the double blinded randomized
controlled trial, or RCT.
Let's say we want to test a drug, which is supposed
to lower blood pressure in those who suffer from hypertension. The researchers
have decided to enroll, say, 100 "subjects". That's what we typically
call the people who are kind enough to play guinea pig in our studies.
The researchers will first do a
randomization of subjects into one of two groups (very often it is more than
one group, but to keep it simple we will assume just two groups). What we mean
with randomization is that we randomly assign each subject to one of the two
groups. One group - the intervention group - will receive the drug, the other
group - the control group - won't. What they get instead is a
sugar pill, a placebo.
With the randomization we want to make sure that, at the
start, or baseline, both groups are indistinguishable from each other with
respect to their average vital parameters. For example, if we were to calculate
the mean age, blood pressure and any other variable for each group, these mean
values would be not different between groups. That's important, because we want
to isolate the effect of the drug. We don't want to worry at the end whether
the effect, or lack thereof, was maybe due to some significant difference
between the groups at baseline.
Once the randomization is done, we organize the
trial in such a way that neither the "subjects" nor their physicians
and nurses know whether they get the placebo or the active drug. Both sides are
blind to what they get and give, which is why this set-up is called
double-blinded. That's an important feature, because a researcher often goes
into a study with a certain expectation of its outcome. Either that outcome
supports his hypothesis, or it doesn't. To eliminate the risk of, more or less
subconsciously, influencing the study towards a desired outcome,
double-blinding is very effective tool.
Fast forward to the end of our trial. We have now
all the data in hand to compare the two groups. After unblinding, the
researchers will compare the two groups with each other. In our example, they
will compare the average, or mean, of the blood pressure values of all the
individuals for each group. If the intervention group's mean value is lower
than that of the control group, then it is plausible to reject the
null-hypothesis, that is to REJECT the idea that the drug is NOT as ineffective
as the placebo (we are, of course, assuming here that the sugar pill didn't
lower the blood pressure of the control group). There are statistical tools to determine whether the difference between the groups may just be a chance event, or whether chance is a very unlikely explanation. We can never rule out chance completely. Now, when we are confident that it is the drug and not pure chance, which has lowered the mean blood pressure in the intervention group, we write our paper to present it in one of the medical journals.
If the subject is a little more sexy, than just lowering blood pressure, there will sure be some journalists who pick it up and report to their readers that, say, eating chocolate makes you slimmer. I'm not kidding. This headline very conveniently went through the media shortly before Easter this year [1]. Good for Hershey who are running it of course on their webpage. And in the media it reads like it did in the Irish Times: "Good news for chocoholics this Easter. Medical Matters: No need for guilt over all those Easter eggs."
I'm not going to comment on the media geniuses, because it's their job to put an angle on every story, so that YOU find it interesting and
read their stuff. But since I'm sure you'll follow these links, just let me
warn you: the chocolate study was an observational study, not an RCT. And one
thing we MUST NOT do with the results of observational studies is to confuse
association with causation. Only when we conduct an RCT, where the intervention
group eats chocolate and the control group doesn't, might we be able to
determine whether there is a causal link. And for obvious reasons we can't
blind the subjects, to whether they eat chocolate or not. But I'm digressing.
Back to our blood pressure study. When we compare the group
averages, everything looks very convincing. And sure enough, as researchers we
are happy with the results, and we are perfectly correct, when we conclude,
that this medicine does its job.
But will it do it for you? When you are
hypertensive? You might be wrong if you say "Yes". And you will be
wrong more often than we, as researchers, or your doctors care to admit. For
one simple reason: The variability of effect within the group. You give 50
people the same drug, and I bet with you, and I'm not the betting type, that
you'll have 50 different results.
The mean value of the entire group glosses
over these inter-individual differences. Let me give you an example from a study
performed on 35 overweight men, who were studied in a supervised and carefully
calculated 12-weeks exercise program, with the intention of reducing body
weight. The mean weight loss was 3.7 kg. That was almost exactly the amount of
weight loss which the researchers had expected from the additional energy
expenditure of the exercise program. But when they looked at each individual, it
became clear that the group mean doesn't tell you anything about how YOU would
fare in that program.
First of all, the standard deviation was 3.6 kg. Now, a
standard deviation of 3.6 kg simply tells you that approximately two thirds of
the participants experienced a weight loss anywhere between 3,7 kg (the mean) minus 3.6 kg
and 3.7kg + 3.6 kg, that is between 0.1 kg and 7.3 kg! That's a lot of kilos. And what about the
remaining one third of those participants? They are even further from these
values. In this case the greatest loser went down by 14 kg, and the biggest
"winner" gained almost 2 kg. A spread of 16 kilos!
Here is the graph which shows you the change on body weight
and fat for each individual participant. Which one would you be?
This effect is what you do not see when you don't read the
studies. And in most studies, it isn't made obvious either.
Which is why, you shouldn't
be surprised to learn that most major drugs are effective only in 25-60% of their
users [2]. The same goes for weight
loss drugs and interventions, for almost everything we study in biomedicine.
That's
not a problem for us in public health. Because a drug, which works in 60% of the
patients, helps us reduce the burden of disease in our population. Public
health is not interested whether you are one of the 60% or not. But you are. And
that's why I believe not only medicine, but also prevention must be individualized.
Which is why the GPS to chronic health, which I currently
develop, is all about helping you find your individual path to your health
objectives.
Why not have a look at it, and maybe even try it out?
References
Print
PDF