13:38
TED2017

Noriko Arai: Can a robot pass a university entrance exam?

Filmed:

Meet Todai Robot, an AI project that performed in the top 20 percent of students on the entrance exam for the University of Tokyo -- without actually understanding a thing. While it's not matriculating anytime soon, Todai Robot's success raises alarming questions for the future of human education. How can we help kids excel at the things that humans will always do better than AI?

- AI expert
Could an AI pass the entrance exam for the University of Tokyo? Noriko Arai oversees a project that wants to find out. Full bio

Today, I'm going to talk about AI and us.
00:13
AI researchers have always said
00:18
that we humans do not need to worry,
00:20
because only menial jobs
will be taken over by machines.
00:22
Is that really true?
00:27
They have also said
that AI will create new jobs,
00:30
so those who lose their jobs
will find a new one.
00:34
Of course.
00:38
But the real question is:
00:39
How many of those
who may lose their jobs to AI
00:41
will be able to land a new one,
00:45
especially when AI is smart enough
to learn better than most of us?
00:48
Let me ask you a question:
00:55
How many of you think
00:58
that AI will pass the entrance examination
of a top university by 2020?
01:00
Oh, so many. OK.
01:07
So some of you may say, "Of course, yes!"
01:10
Now singularity is the issue.
01:15
And some others may say, "Maybe,
01:18
because AI already won
against a top Go player."
01:21
And others may say, "No, never. Uh-uh."
01:27
That means we do not know
the answer yet, right?
01:32
So that was the reason why
I started Todai Robot Project,
01:36
making an AI which passes
the entrance examination
01:41
of the University of Tokyo,
01:45
the top university in Japan.
01:47
This is our Todai Robot.
01:51
And, of course, the brain of the robot
is working in the remote server.
01:56
It is now writing a 600-word essay
02:02
on maritime trade in the 17th century.
02:06
How does that sound?
02:11
Why did I take the entrance exam
as its benchmark?
02:14
Because I thought we had to study
the performance of AI
02:19
in comparison to humans,
02:23
especially on the scales and expertise
02:26
which are believed
to be acquired only by humans
02:28
and only through education.
02:32
To enter Todai, the University of Tokyo,
02:35
you have to pass
two different types of exams.
02:39
The first one is
a national standardized test
02:44
in multiple-choice style.
02:48
You have to take seven subjects
02:50
and achieve a high score --
02:52
I would say like an 84 percent
or more accuracy rate --
02:54
to be allowed to take
the second stage written test
02:59
prepared by Todai.
03:03
So let me first explain
how modern AI works,
03:06
taking the "Jeopardy!" challenge
as an example.
03:12
Here is a typical "Jeopardy!" question:
03:17
"Mozart's last symphony
shares its name with this planet."
03:20
Interestingly, a "Jeopardy!"
question always asks,
03:26
always ends with "this" something:
03:30
"this" planet, "this" country,
03:33
"this" rock musician, and so on.
03:36
In other words, "Jeopardy!" doesn't ask
many different types of questions,
03:39
but a single type,
03:43
which we call "factoid questions."
03:45
By the way, do you know the answer?
03:48
If you do not know the answer
and if you want to know the answer,
03:53
what would you do?
03:58
You Google, right? Of course.
04:00
Why not?
04:03
But you have to pick appropriate keywords
04:04
like "Mozart," "last"
and "symphony" to search.
04:08
The machine basically does the same.
04:13
Then this Wikipedia page
will be ranked top.
04:16
Then the machine reads the page.
04:21
No, uh-uh.
04:23
Unfortunately, none of the modern AIs,
04:25
including Watson, Siri and Todai Robot,
04:28
is able to read.
04:32
But they are very good
at searching and optimizing.
04:35
It will recognize
04:40
that the keywords "Mozart,"
"last" and "symphony"
04:42
are appearing heavily around here.
04:45
So if it can find a word which is a planet
04:49
and which is co-occurring
with these keywords,
04:54
that must be the answer.
04:57
This is how Watson finds
the answer "Jupiter," in this case.
05:00
Our Todai Robot works similarly,
but a bit smarter
05:08
in answering history yes-no questions,
05:12
like, "'Charlemagne repelled the Magyars.'
Is this sentence true or false?"
05:16
Our robot starts producing
a factoid question,
05:23
like: "Charlemagne repelled
[this person type]" by itself.
05:27
Then, "Avars" but not
"Magyars" is ranked top.
05:32
This sentence is likely to be false.
05:38
Our robot does not read,
does not understand,
05:42
but it is statistically
correct in many cases.
05:48
For the second stage written test,
05:54
it is required to write
a 600-word essay like this one:
05:56
[Discuss the rise and fall
of the maritime trade
06:01
in East and Southeast Asia
in the 17th century ...]
06:04
and as I have shown earlier,
06:06
our robot took the sentences
from the textbooks and Wikipedia,
06:07
combined them together,
06:12
and optimized it to produce an essay
06:14
without understanding a thing.
06:17
(Laughter)
06:20
But surprisingly, it wrote a better essay
06:21
than most of the students.
06:26
(Laughter)
06:28
How about mathematics?
06:30
A fully automatic math-solving machine
06:33
has been a dream
06:36
since the birth of the word
"artificial intelligence,"
06:38
but it has stayed at the level
of arithmetic for a long, long time.
06:43
Last year, we finally succeeded
in developing a system
06:51
which solved pre-university-level
problems from end to end,
06:56
like this one.
07:02
This is the original problem
written in Japanese,
07:05
and we had to teach it
2,000 mathematical axioms
07:09
and 8,000 Japanese words
07:14
to make it accept the problems
written in natural language.
07:16
And it is now translating
the original problems
07:22
into machine-readable formulas.
07:25
Weird, but it is now ready
to solve it, I think.
07:30
Go and solve it.
07:36
Yes! It is now executing
symbolic computation.
07:38
Even more weird,
07:44
but probably this is the most
fun part for the machine.
07:45
(Laughter)
07:50
Now it outputs a perfect answer,
07:52
though its proof is impossible to read,
even for mathematicians.
07:55
Anyway, last year our robot
was among the top one percent
08:02
in the second stage written
exam in mathematics.
08:10
(Applause)
08:14
Thank you.
08:18
So, did it enter Todai?
08:19
No, not as I expected.
08:22
Why?
08:26
Because it doesn't understand any meaning.
08:28
Let me show you a typical error
it made in the English test.
08:32
[Nate: We're almost at the bookstore.
Just a few more minutes.
08:36
Sunil: Wait. ______ .
Nate: Thank you! That always happens ...]
08:39
Two people are talking.
08:42
For us, who can understand
the situation --
08:43
[1. "We walked for a long time."
2. "We're almost there."
08:45
3. "Your shoes look expensive."
4. "Your shoelace is untied."]
08:48
it is obvious number four
is the correct answer, right?
08:51
But Todai Robot chose number two,
08:54
even after learning 15 billion
English sentences
08:56
using deep learning technologies.
09:02
OK, so now you might
understand what I said:
09:07
modern AIs do not read,
09:12
do not understand.
09:15
They only disguise as if they do.
09:17
This is the distribution graph
09:24
of half a million students
who took the same exam as Todai Robot.
09:27
Now our Todai Robot
is among the top 20 percent,
09:34
and it was capable to pass
09:40
more than 60 percent
of the universities in Japan --
09:43
but not Todai.
09:47
But see how it is beyond the volume zone
09:50
of to-be white-collar workers.
09:54
You might think I was delighted.
10:00
After all, my robot was surpassing
students everywhere.
10:03
Instead, I was alarmed.
10:09
How on earth could this unintelligent
machine outperform students --
10:13
our children?
10:18
Right?
10:20
I decided to investigate
what was going on in the human world.
10:22
I took hundreds of sentences
from high school textbooks
10:28
and made easy multiple-choice quizzes,
10:33
and asked thousands
of high school students to answer.
10:37
Here is an example:
10:42
[Buddhism spread to ... ,
Christianity to ... and Oceania,
10:43
and Islam to ...]
10:46
Of course, the original problems
are written in Japanese,
10:47
their mother tongue.
10:50
[ ______ has spread to Oceania.
10:51
1. Hinduism 2. Christianity
3. Islam 4. Buddhism ]
10:53
Obviously, Christianity
is the answer, isn't it?
10:55
It's written!
10:58
And Todai Robot chose
the correct answer, too.
11:01
But one-third of junior
high school students
11:06
failed to answer this question.
11:11
Do you think it is only the case in Japan?
11:16
I do not think so,
11:19
because Japan is always ranked
among the top in OECD PISA tests,
11:21
measuring 15-year-old
students' performance in mathematics,
11:28
science and reading
11:31
every three years.
11:33
We have been believing
11:39
that everybody can learn
11:41
and learn well,
11:43
as long as we provide
good learning materials
11:45
free on the web
11:49
so that they can access
through the internet.
11:50
But such wonderful materials
may benefit only those who can read well,
11:53
and the percentage
of those who can read well
12:00
may be much less than we expected.
12:04
How we humans will coexist with AI
12:10
is something we have
to think about carefully,
12:14
based on solid evidence.
12:17
At the same time,
we have to think in a hurry
12:21
because time is running out.
12:25
Thank you.
12:28
(Applause)
12:29
Chris Anderson: Noriko, thank you.
12:34
Noriko Arai: Thank you.
12:36
CA: In your talk, you so beautifully
give us a sense of how AIs think,
12:38
what they can do amazingly
12:43
and what they can't do.
12:45
But -- do I read you right,
12:46
that you think we really need
quite an urgent revolution in education
12:48
to help kids do the things
that humans can do better than AIs?
12:53
NA: Yes, yes, yes.
12:57
Because we humans
can understand the meaning.
12:59
That is something
which is very, very lacking in AI.
13:03
But most of the students
just pack the knowledge
13:08
without understanding
the meaning of the knowledge,
13:12
so that is not knowledge,
that is just memorizing,
13:16
and AI can do the same thing.
13:19
So we have to think about
a new type of education.
13:21
CA: A shift from knowledge,
rote knowledge, to meaning.
13:25
NA: Mm-hmm.
13:28
CA: Well, there's a challenge
for the educators. Thank you so much.
13:29
NA: Thank you very much. Thank you.
13:33
(Applause)
13:34

▲Back to top

About the Speaker:

Noriko Arai - AI expert
Could an AI pass the entrance exam for the University of Tokyo? Noriko Arai oversees a project that wants to find out.

Why you should listen

Noriko Arai is the program director of an AI challenge, Todai Robot Project, which asks the question: Can AI get into the University of Tokyo? The project aims to visualize both the possibilities and the limitation of current AI by setting a concrete goal: a software system that can pass university entrance exams. In 2015 and 2016, Todai Robot achieved top 20 percent in the exams, and passed more than 70 percent of the universities in Japan.

The inventor of Reading Skill Test, in 2017 Arai conducted a large-scale survey on reading skills of high and junior high school students with Japan's Ministry of Education. The results revealed that more than half of junior high school students fail to comprehend sentences sampled from their textbooks. Arai founded the Research Institute of Science for Education to elucidate why so many students fail to read and how she can support them.

More profile about the speaker
Noriko Arai | Speaker | TED.com