Alan Smith: Why we're so bad at statistics
Think you're good at guessing stats? Guess again. Whether we consider ourselves math people or not, our ability to understand and work with numbers is terribly limited, says data visualization expert Alan Smith. In this delightful talk, Smith explores the mismatch between what we know and what we think we know.
Data visualisation editor
Alan Smith uses interactive graphics and statistics to breathe new life into how data is presented.
Double-click the English transcript below to play the video.
Back in 2003,
the UK government carried out a survey.
And it was a survey that measured
levels of numeracy http://e-vid.net/v/en/2674-2
in the population.
And they were shocked to find out
that for every 100 working age
adults in the country, http://e-vid.net/v/en/2674-5
47 of them lacked Level 1 numeracy skills.
Now, Level 1 numeracy skills --
that's low-end GCSE score. http://e-vid.net/v/en/2674-7
It's the ability to deal with fractions,
percentages and decimals. http://e-vid.net/v/en/2674-8
So this figure prompted
a lot of hand-wringing in Whitehall. http://e-vid.net/v/en/2674-9
Policies were changed,
investments were made,
and then they ran
the survey again in 2011. http://e-vid.net/v/en/2674-12
So can you guess
what happened to this number? http://e-vid.net/v/en/2674-13
It went up to 49.
And in fact, when I reported
this figure in the FT, http://e-vid.net/v/en/2674-16
one of our readers joked and said,
"This figure is only shocking
to 51 percent of the population." http://e-vid.net/v/en/2674-18
But I preferred, actually,
the reaction of a schoolchild http://e-vid.net/v/en/2674-20
when I presented
at a school this information, http://e-vid.net/v/en/2674-21
who raised their hand and said,
"How do we know that the person
who made that number http://e-vid.net/v/en/2674-23
isn't one of the 49 percent either?"
So clearly, there's a numeracy issue,
because these are
important skills for life, http://e-vid.net/v/en/2674-27
and a lot of the changes
that we want to introduce in this century http://e-vid.net/v/en/2674-28
involve us becoming
more comfortable with numbers. http://e-vid.net/v/en/2674-29
Now, it's not just an English problem.
OECD this year released some figures
looking at numeracy in young people, http://e-vid.net/v/en/2674-31
and leading the way, the USA --
nearly 40 percent of young people
in the US have low numeracy. http://e-vid.net/v/en/2674-33
Now, England is there too,
but there are seven OECD countries
with figures above 20 percent. http://e-vid.net/v/en/2674-35
That is a problem,
because it doesn't have to be that way. http://e-vid.net/v/en/2674-36
If you look at the far end of this graph,
you can see the Netherlands and Korea
are in single figures. http://e-vid.net/v/en/2674-38
So there's definitely a numeracy
problem that we want to address. http://e-vid.net/v/en/2674-39
Now, as useful as studies like these are,
I think we risk herding people
inadvertently into one of two categories; http://e-vid.net/v/en/2674-41
that there are two kinds of people:
those people that are comfortable
with numbers, that can do numbers, http://e-vid.net/v/en/2674-43
and the people who can't.
And what I'm trying
to talk about here today http://e-vid.net/v/en/2674-45
is to say that I believe
that is a false dichotomy. http://e-vid.net/v/en/2674-46
It's not an immutable pairing.
I think you don't have to have
tremendously high levels of numeracy http://e-vid.net/v/en/2674-48
to be inspired by numbers,
and that should be the starting point
to the journey ahead. http://e-vid.net/v/en/2674-50
And one of the ways in which
we can begin that journey, for me, http://e-vid.net/v/en/2674-51
is looking at statistics.
Now, I am the first to acknowledge
that statistics has got somewhat http://e-vid.net/v/en/2674-53
of an image problem.
It's the part of mathematics
that even mathematicians
don't particularly like, http://e-vid.net/v/en/2674-57
because whereas the rest of maths
is all about precision and certainty, http://e-vid.net/v/en/2674-58
statistics is almost the reverse of that.
But actually, I was a late convert
to the world of statistics myself. http://e-vid.net/v/en/2674-60
If you'd asked my undergraduate professors
what two subjects would I be least likely
to excel in after university, http://e-vid.net/v/en/2674-62
they'd have told you statistics
and computer programming, http://e-vid.net/v/en/2674-63
and yet here I am, about to show you
some statistical graphics http://e-vid.net/v/en/2674-64
that I programmed.
So what inspired that change in me?
What made me think that statistics
was actually an interesting thing? http://e-vid.net/v/en/2674-67
It's really because
statistics are about us. http://e-vid.net/v/en/2674-68
If you look at the etymology
of the word statistics, http://e-vid.net/v/en/2674-69
it's the science of dealing with data
about the state or the community
that we live in. http://e-vid.net/v/en/2674-71
So statistics are about us as a group,
not us as individuals.
And I think as social animals,
we share this fascination about how
we as individuals relate to our groups, http://e-vid.net/v/en/2674-75
to our peers.
And statistics in this way
are at their most powerful http://e-vid.net/v/en/2674-77
when they surprise us.
And there's been some really wonderful
surveys carried out recently http://e-vid.net/v/en/2674-79
by Ipsos MORI in the last few years.
They did a survey of over
1,000 adults in the UK, http://e-vid.net/v/en/2674-81
and said, for every 100 people
in England and Wales, http://e-vid.net/v/en/2674-82
how many of them are Muslim?
Now the average answer from this survey,
which was supposed to be representative
of the total population, was 24. http://e-vid.net/v/en/2674-85
That's what people thought.
British people think 24 out of every 100
people in the country are Muslim. http://e-vid.net/v/en/2674-87
Now, official figures reveal
that figure to be about five. http://e-vid.net/v/en/2674-88
So there's this big variation
between what we think, our perception, http://e-vid.net/v/en/2674-89
and the reality as given by statistics.
And I think that's interesting.
What could possibly be causing
that misperception? http://e-vid.net/v/en/2674-92
And I was so thrilled with this study,
I started to take questions out
in presentations. I was referring to it. http://e-vid.net/v/en/2674-94
Now, I did a presentation
at St. Paul's School for Girls
in Hammersmith, http://e-vid.net/v/en/2674-96
and I had an audience rather like this,
except it was comprised entirely
of sixth-form girls. http://e-vid.net/v/en/2674-98
And I said, "Girls,
how many teenage girls do you think
the British public think http://e-vid.net/v/en/2674-100
get pregnant every year?"
And the girls were apoplectic when I said
the British public think that 15
out of every 100 teenage girls http://e-vid.net/v/en/2674-103
get pregnant in the year.
And they had every right to be angry,
because in fact, I'd have to have
closer to 200 dots http://e-vid.net/v/en/2674-106
before I could color one in,
in terms of what
the official figures tell us. http://e-vid.net/v/en/2674-108
And rather like numeracy,
this is not just an English problem. http://e-vid.net/v/en/2674-109
Ipsos MORI expanded the survey
in recent years to go across the world. http://e-vid.net/v/en/2674-110
And so, they asked Saudi Arabians,
for every 100 adults in your country,
how many of them are overweight or obese?
And the average answer from the Saudis
was just over a quarter. http://e-vid.net/v/en/2674-114
That's what they thought.
Just over a quarter of adults
are overweight or obese. http://e-vid.net/v/en/2674-116
The official figures show, actually,
it's nearer to three-quarters. http://e-vid.net/v/en/2674-117
So again, a big variation.
And I love this one: they asked in Japan,
they asked the Japanese, http://e-vid.net/v/en/2674-120
for every 100 Japanese people,
how many of them live in rural areas?
The average was about a 50-50 split,
just over halfway. http://e-vid.net/v/en/2674-123
They thought 56 out of every 100
Japanese people lived in rural areas. http://e-vid.net/v/en/2674-124
The official figure is seven.
So extraordinary variations,
and surprising to some, http://e-vid.net/v/en/2674-126
but not surprising to people
who have read the work http://e-vid.net/v/en/2674-127
of Daniel Kahneman, for example,
the Nobel-winning economist. http://e-vid.net/v/en/2674-128
He and his colleague, Amos Tversky,
spent years researching this disjoint http://e-vid.net/v/en/2674-129
between what people perceive
and the reality, http://e-vid.net/v/en/2674-130
the fact that people are actually
pretty poor intuitive statisticians. http://e-vid.net/v/en/2674-131
And there are many reasons for this.
Individual experiences, certainly,
can influence our perceptions, http://e-vid.net/v/en/2674-133
but so, too, can things like the media
reporting things by exception, http://e-vid.net/v/en/2674-134
rather than what's normal.
Kahneman had a nice way
of referring to that. http://e-vid.net/v/en/2674-136
He said, "We can be blind
to the obvious" -- http://e-vid.net/v/en/2674-137
so we've got the numbers wrong --
"but we can be blind
to our blindness about it." http://e-vid.net/v/en/2674-139
And that has enormous
repercussions for decision making. http://e-vid.net/v/en/2674-140
So at the statistics office
while this was all going on, http://e-vid.net/v/en/2674-141
I thought this was really interesting.
I said, this is clearly a global problem,
but maybe geography is the issue here.
These were questions that were all about,
how well do you know your country? http://e-vid.net/v/en/2674-145
So in this case, it's how well
do you know 64 million people? http://e-vid.net/v/en/2674-146
Not very well, it turns out.
I can't do that. http://e-vid.net/v/en/2674-147
So I had an idea,
which was to think about
this same sort of approach http://e-vid.net/v/en/2674-149
but to think about it
in a very local sense. http://e-vid.net/v/en/2674-150
Is this a local?
If we reframe the questions and say,
how well do you know your local area,
would your answers be any more accurate?
So I devised a quiz:
How well do you know your area?
It's a simple Web app.
You put in a post code
and then it will ask you questions
based on census data http://e-vid.net/v/en/2674-159
for your local area.
And I was very conscious
in designing this. http://e-vid.net/v/en/2674-161
I wanted to make it open
to the widest possible range of people, http://e-vid.net/v/en/2674-162
not just the 49 percent
who can get the numbers. http://e-vid.net/v/en/2674-163
I wanted everyone to engage with it.
So for the design of the quiz,
I was inspired by the isotypes
of Otto Neurath from the 1920s and '30s.
Now, these are methods
for representing numbers http://e-vid.net/v/en/2674-168
using repeating icons.
And the numbers are there,
but they sit in the background. http://e-vid.net/v/en/2674-170
So it's a great way
of representing quantity http://e-vid.net/v/en/2674-171
without resorting to using terms
like "percentage," http://e-vid.net/v/en/2674-172
"fractions" and "ratios."
So here's the quiz.
The layout of the quiz is,
you have your repeating icons
on the left-hand side there, http://e-vid.net/v/en/2674-176
and a map showing you the area
we're asking you questions about http://e-vid.net/v/en/2674-177
on the right-hand side.
There are seven questions.
Each question, there's a possible answer
between zero and a hundred, http://e-vid.net/v/en/2674-180
and at the end of the quiz,
you get an overall score
between zero and a hundred. http://e-vid.net/v/en/2674-182
And so because this is TEDxExeter,
I thought we would have
a quick look at the quiz http://e-vid.net/v/en/2674-184
for the first few questions of Exeter.
And so the first question is:
For every 100 people,
how many are aged under 16? http://e-vid.net/v/en/2674-187
Now, I don't know Exeter very well
at all, so I had a guess at this, http://e-vid.net/v/en/2674-188
but it gives you an idea
of how this quiz works. http://e-vid.net/v/en/2674-189
You drag the slider
to highlight your icons, http://e-vid.net/v/en/2674-190
and then just click "Submit" to answer,
and we animate away the difference
between your answer and reality. http://e-vid.net/v/en/2674-192
And it turns out, I was a pretty
terrible guess: five. http://e-vid.net/v/en/2674-193
How about the next question?
This is asking about
what the average age is, http://e-vid.net/v/en/2674-195
so the age at which half
the population are younger http://e-vid.net/v/en/2674-196
and half the population are older.
And I thought 35 -- that sounds
middle-aged to me. http://e-vid.net/v/en/2674-198
Actually, in Exeter,
it's incredibly young, http://e-vid.net/v/en/2674-200
and I had underestimated the impact
of the university in this area. http://e-vid.net/v/en/2674-201
The questions get harder
as you go through. http://e-vid.net/v/en/2674-202
So this one's now asking
about homeownership: http://e-vid.net/v/en/2674-203
For every 100 households, how many
are owned with a mortgage or loan? http://e-vid.net/v/en/2674-204
And I hedged my bets here,
because I didn't want to be
more than 50 out on the answer. http://e-vid.net/v/en/2674-206
And actually, these get harder,
these questions, http://e-vid.net/v/en/2674-208
because when you're in an area,
when you're in a community, http://e-vid.net/v/en/2674-209
things like age -- there are clues
to whether a population is old or young. http://e-vid.net/v/en/2674-210
Just by looking around
the area, you can see it. http://e-vid.net/v/en/2674-211
Something like homeownership
is much more difficult to see, http://e-vid.net/v/en/2674-212
so we revert to our own heuristics,
our own biases about how many people
we think own their own homes. http://e-vid.net/v/en/2674-214
Now the truth is,
when we published this quiz, http://e-vid.net/v/en/2674-215
the census data that it's based on
was already a few years old. http://e-vid.net/v/en/2674-216
We've had online applications
that allow you to put in a post code http://e-vid.net/v/en/2674-217
and get statistics back for years.
So in some senses,
this was all a little bit old
and not necessarily new. http://e-vid.net/v/en/2674-220
But I was interested to see
what reaction we might get http://e-vid.net/v/en/2674-221
by game-ifying the data
in the way that we have, http://e-vid.net/v/en/2674-222
by using animation
and playing on the fact
that people have their own preconceptions. http://e-vid.net/v/en/2674-224
It turns out, the reaction was, um ...
was more than I could have hoped for.
It was a long-held ambition of mine
to bring down a statistics website http://e-vid.net/v/en/2674-227
due to public demand.
This URL contains the words
"statistics," "gov" and "UK," http://e-vid.net/v/en/2674-230
which are three of people's least
favorite words in a URL. http://e-vid.net/v/en/2674-231
And the amazing thing about this
was that the website came down http://e-vid.net/v/en/2674-232
at quarter to 10 at night,
because people were actually
engaging with this data http://e-vid.net/v/en/2674-234
of their own free will,
using their own personal time.
I was very interested to see
that we got something like
a quarter of a million people http://e-vid.net/v/en/2674-238
playing the quiz within the space
of 48 hours of launching it. http://e-vid.net/v/en/2674-239
And it sparked an enormous discussion
online, on social media, http://e-vid.net/v/en/2674-240
which was largely dominated
by people having fun
with their misconceptions, http://e-vid.net/v/en/2674-242
which is something that
I couldn't have hoped for any better, http://e-vid.net/v/en/2674-243
in some respects.
I also liked the fact that people started
sending it to politicians. http://e-vid.net/v/en/2674-245
How well do you know the area
you claim to represent? http://e-vid.net/v/en/2674-246
And then just to finish,
going back to the two kinds of people,
I thought it would be
really interesting to see http://e-vid.net/v/en/2674-250
how people who are good with numbers
would do on this quiz. http://e-vid.net/v/en/2674-251
The national statistician
of England and Wales, John Pullinger, http://e-vid.net/v/en/2674-252
you would expect he would be pretty good.
He got 44 for his own area.
Jeremy Paxman -- admittedly,
after a glass of wine -- 36. http://e-vid.net/v/en/2674-256
It just shows you that the numbers
can inspire us all. http://e-vid.net/v/en/2674-258
They can surprise us all.
So very often, we talk about statistics
as being the science of uncertainty.
My parting thought for today is:
actually, statistics is the science of us.
And that's why we should
be fascinated by numbers. http://e-vid.net/v/en/2674-264
Thank you very much.
http://e-vid.net/v/en/2674-266 ▲Back to top About the Speaker: Alan Smith
Data visualisation editor
Alan Smith uses interactive graphics and statistics to breathe new life into how data is presented.
Why you should listen
Alan Smith is Data Visualisation Editor at the Financial Times in London. Previously he was Head of Digital Content at the UK Office for National Statistics (ONS).
With a background in cartography and digital mapping, he has spent the last decade finding ways of bringing statistics to wider audiences. In 2010, he was an inaugural recipient of the Royal Statistical Society's Award for Excellence in Official Statistics. He was appointed Office of the Order of the British Empire (OBE) in the Queen's 2011 Birthday Honours list.
More profile about the speaker Alan Smith | Speaker | TED.com The original video on TED.com: