sponsored links
TEDGlobal 2011

Ben Goldacre: Battling bad science

July 13, 2011

Every day there are news reports of new health advice, but how can you know if they're right? Doctor and epidemiologist Ben Goldacre shows us, at high speed, the ways evidence can be distorted, from the blindingly obvious nutrition claims to the very subtle tricks of the pharmaceutical industry.

Ben Goldacre - Debunker
Ben Goldacre unpicks dodgy scientific claims made by scaremongering journalists, dubious government reports, pharmaceutical corporations, PR companies and quacks. Full bio

sponsored links
Double-click the English subtitles below to play the video.
So I'm a doctor, but I kind of slipped sideways into research,
00:15
and now I'm an epidemiologist.
00:18
And nobody really knows what epidemiology is.
00:20
Epidemiology is the science of how we know in the real world
00:22
if something is good for you or bad for you.
00:25
And it's best understood through example
00:27
as the science of those crazy, wacky newspaper headlines.
00:29
And these are just some of the examples.
00:34
These are from the Daily Mail. Every country in the world has a newspaper like this.
00:36
It has this bizarre, ongoing philosophical project
00:39
of dividing all the inanimate objects in the world
00:42
into the ones that either cause or prevent cancer.
00:44
So here are some of the things they said cause cancer recently:
00:47
divorce, Wi-Fi, toiletries and coffee.
00:49
Here are some of the things they say prevents cancer:
00:51
crusts, red pepper, licorice and coffee.
00:53
So already you can see there are contradictions.
00:55
Coffee both causes and prevents cancer.
00:57
And as you start to read on, you can see
00:59
that maybe there's some kind of political valence behind some of this.
01:01
So for women, housework prevents breast cancer,
01:04
but for men, shopping could make you impotent.
01:06
So we know that we need to start
01:09
unpicking the science behind this.
01:12
And what I hope to show
01:15
is that unpicking dodgy claims,
01:17
unpicking the evidence behind dodgy claims,
01:19
isn't a kind of nasty carping activity;
01:21
it's socially useful,
01:24
but it's also an extremely valuable
01:26
explanatory tool.
01:28
Because real science is all about
01:30
critically appraising the evidence for somebody else's position.
01:32
That's what happens in academic journals.
01:34
That's what happens at academic conferences.
01:36
The Q&A session after a post-op presents data
01:38
is often a blood bath.
01:40
And nobody minds that. We actively welcome it.
01:42
It's like a consenting intellectual S&M activity.
01:44
So what I'm going to show you
01:47
is all of the main things,
01:49
all of the main features of my discipline --
01:51
evidence-based medicine.
01:53
And I will talk you through all of these
01:55
and demonstrate how they work,
01:57
exclusively using examples of people getting stuff wrong.
01:59
So we'll start with the absolute weakest form of evidence known to man,
02:02
and that is authority.
02:05
In science, we don't care how many letters you have after your name.
02:07
In science, we want to know what your reasons are for believing something.
02:10
How do you know that something is good for us
02:13
or bad for us?
02:15
But we're also unimpressed by authority,
02:17
because it's so easy to contrive.
02:19
This is somebody called Dr. Gillian McKeith Ph.D,
02:21
or, to give her full medical title, Gillian McKeith.
02:23
(Laughter)
02:26
Again, every country has somebody like this.
02:29
She is our TV diet guru.
02:31
She has massive five series of prime-time television,
02:33
giving out very lavish and exotic health advice.
02:36
She, it turns out, has a non-accredited correspondence course Ph.D.
02:39
from somewhere in America.
02:42
She also boasts that she's a certified professional member
02:44
of the American Association of Nutritional Consultants,
02:46
which sounds very glamorous and exciting.
02:48
You get a certificate and everything.
02:50
This one belongs to my dead cat Hetti. She was a horrible cat.
02:52
You just go to the website, fill out the form,
02:54
give them $60, and it arrives in the post.
02:56
Now that's not the only reason that we think this person is an idiot.
02:58
She also goes and says things like,
03:00
you should eat lots of dark green leaves,
03:02
because they contain lots of chlorophyll, and that will really oxygenate your blood.
03:04
And anybody who's done school biology remembers
03:06
that chlorophyll and chloroplasts
03:08
only make oxygen in sunlight,
03:10
and it's quite dark in your bowels after you've eaten spinach.
03:12
Next, we need proper science, proper evidence.
03:15
So, "Red wine can help prevent breast cancer."
03:18
This is a headline from the Daily Telegraph in the U.K.
03:20
"A glass of red wine a day could help prevent breast cancer."
03:22
So you go and find this paper, and what you find
03:25
is it is a real piece of science.
03:27
It is a description of the changes in one enzyme
03:29
when you drip a chemical extracted from some red grape skin
03:32
onto some cancer cells
03:35
in a dish on a bench in a laboratory somewhere.
03:37
And that's a really useful thing to describe
03:40
in a scientific paper,
03:42
but on the question of your own personal risk
03:44
of getting breast cancer if you drink red wine,
03:46
it tells you absolutely bugger all.
03:48
Actually, it turns out that your risk of breast cancer
03:50
actually increases slightly
03:52
with every amount of alcohol that you drink.
03:54
So what we want is studies in real human people.
03:56
And here's another example.
04:00
This is from Britain's leading diet and nutritionist in the Daily Mirror,
04:02
which is our second biggest selling newspaper.
04:05
"An Australian study in 2001
04:07
found that olive oil in combination with fruits, vegetables and pulses
04:09
offers measurable protection against skin wrinklings."
04:11
And then they give you advice:
04:13
"If you eat olive oil and vegetables, you'll have fewer skin wrinkles."
04:15
And they very helpfully tell you how to go and find the paper.
04:17
So you go and find the paper, and what you find is an observational study.
04:19
Obviously nobody has been able
04:22
to go back to 1930,
04:24
get all the people born in one maternity unit,
04:26
and half of them eat lots of fruit and veg and olive oil,
04:29
and then half of them eat McDonald's,
04:31
and then we see how many wrinkles you've got later.
04:33
You have to take a snapshot of how people are now.
04:35
And what you find is, of course,
04:37
people who eat veg and olive oil have fewer skin wrinkles.
04:39
But that's because people who eat fruit and veg and olive oil,
04:42
they're freaks, they're not normal, they're like you;
04:45
they come to events like this.
04:48
They are posh, they're wealthy, they're less likely to have outdoor jobs,
04:50
they're less likely to do manual labor,
04:53
they have better social support, they're less likely to smoke --
04:55
so for a whole host of fascinating, interlocking
04:57
social, political and cultural reasons,
04:59
they are less likely to have skin wrinkles.
05:01
That doesn't mean that it's the vegetables or the olive oil.
05:03
(Laughter)
05:05
So ideally what you want to do is a trial.
05:07
And everybody thinks they're very familiar with the idea of a trial.
05:10
Trials are very old. The first trial was in the Bible -- Daniel 1:12.
05:12
It's very straightforward -- you take a bunch of people, you split them in half,
05:15
you treat one group one way, you treat the other group the other way,
05:17
and a little while later, you follow them up
05:19
and see what happened to each of them.
05:21
So I'm going to tell you about one trial,
05:23
which is probably the most well-reported trial
05:25
in the U.K. news media over the past decade.
05:27
And this is the trial of fish oil pills.
05:29
And the claim was fish oil pills improve school performance and behavior
05:31
in mainstream children.
05:33
And they said, "We've done a trial.
05:35
All the previous trials were positive, and we know this one's gonna be too."
05:37
That should always ring alarm bells.
05:39
Because if you already know the answer to your trial, you shouldn't be doing one.
05:41
Either you've rigged it by design,
05:44
or you've got enough data so there's no need to randomize people anymore.
05:46
So this is what they were going to do in their trial.
05:49
They were taking 3,000 children,
05:52
they were going to give them all these huge fish oil pills,
05:54
six of them a day,
05:56
and then a year later, they were going to measure their school exam performance
05:58
and compare their school exam performance
06:01
against what they predicted their exam performance would have been
06:03
if they hadn't had the pills.
06:05
Now can anybody spot a flaw in this design?
06:08
And no professors of clinical trial methodology
06:11
are allowed to answer this question.
06:14
So there's no control; there's no control group.
06:16
But that sounds really techie.
06:18
That's a technical term.
06:20
The kids got the pills, and then their performance improved.
06:22
What else could it possibly be if it wasn't the pills?
06:24
They got older. We all develop over time.
06:27
And of course, also there's the placebo effect.
06:30
The placebo effect is one of the most fascinating things in the whole of medicine.
06:32
It's not just about taking a pill, and your performance and your pain getting better.
06:34
It's about our beliefs and expectations.
06:37
It's about the cultural meaning of a treatment.
06:39
And this has been demonstrated in a whole raft of fascinating studies
06:41
comparing one kind of placebo against another.
06:44
So we know, for example, that two sugar pills a day
06:47
are a more effective treatment for getting rid of gastric ulcers
06:49
than one sugar pill.
06:51
Two sugar pills a day beats one sugar pill a day.
06:53
And that's an outrageous and ridiculous finding, but it's true.
06:55
We know from three different studies on three different types of pain
06:58
that a saltwater injection is a more effective treatment for pain
07:00
than taking a sugar pill, taking a dummy pill that has no medicine in it --
07:03
not because the injection or the pills do anything physically to the body,
07:07
but because an injection feels like a much more dramatic intervention.
07:10
So we know that our beliefs and expectations
07:13
can be manipulated,
07:15
which is why we do trials
07:17
where we control against a placebo --
07:19
where one half of the people get the real treatment
07:21
and the other half get placebo.
07:23
But that's not enough.
07:25
What I've just shown you are examples of the very simple and straightforward ways
07:28
that journalists and food supplement pill peddlers
07:31
and naturopaths
07:33
can distort evidence for their own purposes.
07:35
What I find really fascinating
07:38
is that the pharmaceutical industry
07:40
uses exactly the same kinds of tricks and devices,
07:42
but slightly more sophisticated versions of them,
07:44
in order to distort the evidence that they give to doctors and patients,
07:47
and which we use to make vitally important decisions.
07:50
So firstly, trials against placebo:
07:53
everybody thinks they know that a trial should be
07:55
a comparison of your new drug against placebo.
07:57
But actually in a lot of situations that's wrong.
07:59
Because often we already have a very good treatment that is currently available,
08:01
so we don't want to know that your alternative new treatment
08:04
is better than nothing.
08:06
We want to know that it's better than the best currently available treatment that we have.
08:08
And yet, repeatedly, you consistently see people doing trials
08:11
still against placebo.
08:14
And you can get license to bring your drug to market
08:16
with only data showing that it's better than nothing,
08:18
which is useless for a doctor like me trying to make a decision.
08:20
But that's not the only way you can rig your data.
08:23
You can also rig your data
08:25
by making the thing you compare your new drug against
08:27
really rubbish.
08:29
You can give the competing drug in too low a dose,
08:31
so that people aren't properly treated.
08:33
You can give the competing drug in too high a dose,
08:35
so that people get side effects.
08:37
And this is exactly what happened
08:39
which antipsychotic medication for schizophrenia.
08:41
20 years ago, a new generation of antipsychotic drugs were brought in
08:43
and the promise was that they would have fewer side effects.
08:46
So people set about doing trials of these new drugs
08:49
against the old drugs,
08:51
but they gave the old drugs in ridiculously high doses --
08:53
20 milligrams a day of haloperidol.
08:55
And it's a foregone conclusion,
08:57
if you give a drug at that high a dose,
08:59
that it will have more side effects and that your new drug will look better.
09:01
10 years ago, history repeated itself, interestingly,
09:04
when risperidone, which was the first of the new-generation antipscyhotic drugs,
09:06
came off copyright, so anybody could make copies.
09:09
Everybody wanted to show that their drug was better than risperidone,
09:12
so you see a bunch of trials comparing new antipsychotic drugs
09:14
against risperidone at eight milligrams a day.
09:17
Again, not an insane dose, not an illegal dose,
09:19
but very much at the high end of normal.
09:21
And so you're bound to make your new drug look better.
09:23
And so it's no surprise that overall,
09:26
industry-funded trials
09:29
are four times more likely to give a positive result
09:31
than independently sponsored trials.
09:33
But -- and it's a big but --
09:36
(Laughter)
09:39
it turns out,
09:41
when you look at the methods used by industry-funded trials,
09:43
that they're actually better
09:46
than independently sponsored trials.
09:48
And yet, they always manage to to get the result that they want.
09:50
So how does this work?
09:53
How can we explain this strange phenomenon?
09:55
Well it turns out that what happens
09:58
is the negative data goes missing in action;
10:00
it's withheld from doctors and patients.
10:02
And this is the most important aspect of the whole story.
10:04
It's at the top of the pyramid of evidence.
10:06
We need to have all of the data on a particular treatment
10:08
to know whether or not it really is effective.
10:11
And there are two different ways that you can spot
10:13
whether some data has gone missing in action.
10:15
You can use statistics, or you can use stories.
10:17
I personally prefer statistics, so that's what I'm going to do first.
10:20
This is something called funnel plot.
10:22
And a funnel plot is a very clever way of spotting
10:24
if small negative trials have disappeared, have gone missing in action.
10:26
So this is a graph of all of the trials
10:29
that have been done on a particular treatment.
10:31
And as you go up towards the top of the graph,
10:33
what you see is each dot is a trial.
10:35
And as you go up, those are the bigger trials, so they've got less error in them.
10:37
So they're less likely to be randomly false positives, randomly false negatives.
10:40
So they all cluster together.
10:43
The big trials are closer to the true answer.
10:45
Then as you go further down at the bottom,
10:47
what you can see is, over on this side, the spurious false negatives,
10:49
and over on this side, the spurious false positives.
10:52
If there is publication bias,
10:54
if small negative trials have gone missing in action,
10:56
you can see it on one of these graphs.
10:59
So you can see here that the small negative trials
11:01
that should be on the bottom left have disappeared.
11:03
This is a graph demonstrating the presence of publication bias
11:05
in studies of publication bias.
11:08
And I think that's the funniest epidemiology joke
11:10
that you will ever hear.
11:12
That's how you can prove it statistically,
11:14
but what about stories?
11:16
Well they're heinous, they really are.
11:18
This is a drug called reboxetine.
11:20
This is a drug that I myself have prescribed to patients.
11:22
And I'm a very nerdy doctor.
11:24
I hope I try to go out of my way to try and read and understand all the literature.
11:26
I read the trials on this. They were all positive. They were all well-conducted.
11:29
I found no flaw.
11:32
Unfortunately, it turned out,
11:34
that many of these trials were withheld.
11:36
In fact, 76 percent
11:38
of all of the trials that were done on this drug
11:40
were withheld from doctors and patients.
11:42
Now if you think about it,
11:44
if I tossed a coin a hundred times,
11:46
and I'm allowed to withhold from you
11:48
the answers half the times,
11:50
then I can convince you
11:52
that I have a coin with two heads.
11:54
If we remove half of the data,
11:56
we can never know what the true effect size of these medicines is.
11:58
And this is not an isolated story.
12:01
Around half of all of the trial data on antidepressants has been withheld,
12:03
but it goes way beyond that.
12:07
The Nordic Cochrane Group were trying to get a hold of the data on that
12:09
to bring it all together.
12:11
The Cochrane Groups are an international nonprofit collaboration
12:13
that produce systematic reviews of all of the data that has ever been shown.
12:16
And they need to have access to all of the trial data.
12:19
But the companies withheld that data from them,
12:22
and so did the European Medicines Agency
12:25
for three years.
12:27
This is a problem that is currently lacking a solution.
12:29
And to show how big it goes, this is a drug called Tamiflu,
12:32
which governments around the world
12:35
have spent billions and billions of dollars on.
12:37
And they spend that money on the promise
12:39
that this is a drug which will reduce the rate
12:41
of complications with flu.
12:43
We already have the data
12:45
showing that it reduces the duration of your flu by a few hours.
12:47
But I don't really care about that. Governments don't care about that.
12:49
I'm very sorry if you have the flu, I know it's horrible,
12:51
but we're not going to spend billions of dollars
12:54
trying to reduce the duration of your flu symptoms
12:56
by half a day.
12:58
We prescribe these drugs, we stockpile them for emergencies
13:00
on the understanding that they will reduce the number of complications,
13:02
which means pneumonia and which means death.
13:04
The infectious diseases Cochrane Group, which are based in Italy,
13:07
has been trying to get
13:10
the full data in a usable form out of the drug companies
13:12
so that they can make a full decision
13:15
about whether this drug is effective or not,
13:18
and they've not been able to get that information.
13:20
This is undoubtedly
13:23
the single biggest ethical problem
13:25
facing medicine today.
13:28
We cannot make decisions
13:30
in the absence of all of the information.
13:33
So it's a little bit difficult from there
13:37
to spin in some kind of positive conclusion.
13:40
But I would say this:
13:44
I think that sunlight
13:48
is the best disinfectant.
13:51
All of these things are happening in plain sight,
13:53
and they're all protected
13:56
by a force field of tediousness.
13:58
And I think, with all of the problems in science,
14:01
one of the best things that we can do
14:03
is to lift up the lid,
14:05
finger around in the mechanics and peer in.
14:07
Thank you very much.
14:09
(Applause)
14:11

sponsored links

Ben Goldacre - Debunker
Ben Goldacre unpicks dodgy scientific claims made by scaremongering journalists, dubious government reports, pharmaceutical corporations, PR companies and quacks.

Why you should listen

"It was the MMR story that finally made me crack," begins the Bad Science manifesto, referring to the sensationalized -- and now-refuted -- link between vaccines and autism. With that sentence Ben Goldacre fired the starting shot of a crusade waged from the pages of The Guardian from 2003 to 2011, on an addicitve Twitter feed, and in bestselling books, including Bad Science and his latest, Bad Pharma, which puts the $600 billion global pharmaceutical industry under the microscope. What he reveals is a fascinating, terrifying mess.

Goldacre was trained in medicine at Oxford and London, and works as an academic in epidemiology. Helped along by this inexhaustible supply of material, he also travels the speaking circuit, promoting skepticism and nerdish curiosity with fire, wit, fast delivery and a lovable kind of exasperation. (He might even convince you that real science, sober reporting and reason are going to win in the end.)

As he writes, "If you're a journalist who misrepresents science for the sake of a headline, a politician more interested in spin than evidence, or an advertiser who loves pictures of molecules in little white coats, then beware: your days are numbered."

Read an excerpt of Bad Pharma >>

The original video is available on TED.com
sponsored links

If you need translations, you can install "Google Translate" extension into your Chrome Browser.
Furthermore, you can change playback rate by installing "Video Speed Controller" extension.

Data provided by TED.

This website is owned and operated by Tokyo English Network.
The developer's blog is here.