TED2017
Stuart Russell: 3 principles for creating safer AI
斯图尔特·罗素: 人工智能是如何让我们变得更好的
Filmed:
Readability: 3.9
1,465,832 views
我们应该如何在发挥人工智能最大用途的同时，预防那些机器人可能带来的威胁呢？随着人工智能的日益完善和发展，人工智能先驱斯图尔特·罗素正在创造一些不同的东西：那就是具有无限可能的机器人。让我们听听他对人类该如何兼容人工智能的看法，如何才能真正利用人工智能使其利用常识、利他主义以及人类的价值观来解决问题。
Stuart Russell - AI expert
Stuart Russell wrote the standard text on AI; now he thinks deeply on AI's future -- and the future of us humans, too. Full bio
Stuart Russell wrote the standard text on AI; now he thinks deeply on AI's future -- and the future of us humans, too. Full bio
Double-click the English transcript below to play the video.
这是李世石。
00:12
This is Lee Sedol.
0
712
1552
李世石是全世界
最顶尖的围棋高手之一，
最顶尖的围棋高手之一，
00:14
Lee Sedol is one of the world's
greatest Go players,
greatest Go players,
1
2288
3997
在这一刻，他所经历的
足以让我硅谷的朋友们
足以让我硅谷的朋友们
00:18
and he's having what my friends
in Silicon Valley call
in Silicon Valley call
2
6309
2885
00:21
a "Holy Cow" moment --
3
9218
1510
喊一句”我的天啊“——
00:22
(Laughter)
4
10752
1073
（笑声）
00:23
a moment where we realize
5
11849
2188
在这一刻，我们意识到
原来人工智能发展的进程
比我们预想的要快得多。
比我们预想的要快得多。
00:26
that AI is actually progressing
a lot faster than we expected.
a lot faster than we expected.
6
14061
3296
00:30
So humans have lost on the Go board.
What about the real world?
What about the real world?
7
18154
3047
人们在围棋棋盘上已经输了，
那在现实世界中又如何呢？
那在现实世界中又如何呢？
当然了，现实世界要
比围棋棋盘要大得多，
比围棋棋盘要大得多，
00:33
Well, the real world is much bigger,
8
21225
2100
复杂得多。
00:35
much more complicated than the Go board.
9
23349
2249
相比之下每一步也没那么明确，
00:37
It's a lot less visible,
10
25622
1819
00:39
but it's still a decision problem.
11
27465
2038
但现实世界仍然是一个选择性问题。
00:42
And if we think about some
of the technologies
of the technologies
12
30948
2321
如果我们想想那一些在不久的未来，
即将来临的新科技……
00:45
that are coming down the pike ...
13
33293
1749
00:47
Noriko [Arai] mentioned that reading
is not yet happening in machines,
is not yet happening in machines,
14
35738
4335
Noriko提到机器还不能进行阅读，
至少达不到理解的程度，
00:52
at least with understanding.
15
40097
1500
但这迟早会发生，
00:53
But that will happen,
16
41621
1536
而当它发生时，
00:55
and when that happens,
17
43181
1771
不久之后，
00:56
very soon afterwards,
18
44976
1187
机器就将读遍人类写下的所有东西。
00:58
machines will have read everything
that the human race has ever written.
that the human race has ever written.
19
46187
4572
01:03
And that will enable machines,
20
51850
2030
这将使机器除了拥有
比人类看得更远的能力，
01:05
along with the ability to look
further ahead than humans can,
further ahead than humans can,
21
53904
2920
就像我们在围棋中看到的那样，
01:08
as we've already seen in Go,
22
56848
1680
如果机器能接触到比人类更多的信息，
01:10
if they also have access
to more information,
to more information,
23
58552
2164
则将能够在现实世界中
做出比人类更好的选择。
做出比人类更好的选择。
01:12
they'll be able to make better decisions
in the real world than we can.
in the real world than we can.
24
60740
4268
那这是一件好事吗？
01:18
So is that a good thing?
25
66792
1606
01:21
Well, I hope so.
26
69898
2232
我当然希望如此。
01:26
Our entire civilization,
everything that we value,
everything that we value,
27
74694
3255
人类的全部文明，
我们所珍视的一切，
我们所珍视的一切，
都是基于我们的智慧之上。
01:29
is based on our intelligence.
28
77973
2068
如果我们能掌控更强大的智能，
01:32
And if we had access
to a lot more intelligence,
to a lot more intelligence,
29
80065
3694
那我们人类的 创造力
就真的没有极限了。
就真的没有极限了。
01:35
then there's really no limit
to what the human race can do.
to what the human race can do.
30
83783
3302
01:40
And I think this could be,
as some people have described it,
as some people have described it,
31
88665
3325
我认为这可能就像很多人描述的那样
会成为人类历史上最重要的事件。
01:44
the biggest event in human history.
32
92014
2016
01:48
So why are people saying things like this,
33
96665
2829
那为什么有的人会说出以下的言论，
说人工智能将是人类的末日呢？
01:51
that AI might spell the end
of the human race?
of the human race?
34
99518
2876
01:55
Is this a new thing?
35
103438
1659
这是一个新事物吗？
这只关乎伊隆马斯克、
比尔盖茨，和斯提芬霍金吗？
比尔盖茨，和斯提芬霍金吗？
01:57
Is it just Elon Musk and Bill Gates
and Stephen Hawking?
and Stephen Hawking?
36
105121
4110
02:01
Actually, no. This idea
has been around for a while.
has been around for a while.
37
109953
3262
其实不是的，人工智能
这个概念已经存在很长时间了。
这个概念已经存在很长时间了。
请看这段话：
02:05
Here's a quotation:
38
113239
1962
“即便我们能够将机器
维持在一个屈服于我们的地位，
维持在一个屈服于我们的地位，
02:07
"Even if we could keep the machines
in a subservient position,
in a subservient position,
39
115225
4350
比如说，在战略性时刻将电源关闭。”——
02:11
for instance, by turning off the power
at strategic moments" --
at strategic moments" --
40
119599
2984
我等会儿再来讨论
”关闭电源“这一话题，
”关闭电源“这一话题，
02:14
and I'll come back to that
"turning off the power" idea later on --
"turning off the power" idea later on --
41
122607
3237
”我们，作为一个物种，
仍然应该自感惭愧。“
仍然应该自感惭愧。“
02:17
"we should, as a species,
feel greatly humbled."
feel greatly humbled."
42
125868
2804
02:22
So who said this?
This is Alan Turing in 1951.
This is Alan Turing in 1951.
43
130177
3448
这段话是谁说的呢？
是阿兰图灵，他在1951年说的。
是阿兰图灵，他在1951年说的。
02:26
Alan Turing, as you know,
is the father of computer science
is the father of computer science
44
134300
2763
阿兰图灵，众所皆知，
是计算机科学之父。
是计算机科学之父。
从很多意义上说，
他也是人工智能之父。
他也是人工智能之父。
02:29
and in many ways,
the father of AI as well.
the father of AI as well.
45
137087
3048
02:33
So if we think about this problem,
46
141239
1882
当我们考虑这个问题，
创造一个比自己更智能的
物种的问题时，
物种的问题时，
02:35
the problem of creating something
more intelligent than your own species,
more intelligent than your own species,
47
143145
3787
我们不妨将它称为”大猩猩问题“，
02:38
we might call this "the gorilla problem,"
48
146956
2622
02:42
because gorillas' ancestors did this
a few million years ago,
a few million years ago,
49
150345
3750
因为这正是大猩猩的
祖先们几百万年前所经历的。
祖先们几百万年前所经历的。
我们今天可以去问大猩猩们：
02:46
and now we can ask the gorillas:
50
154119
1745
02:48
Was this a good idea?
51
156752
1160
那么做是不是一个好主意？
在这幅图里，大猩猩们正在
开会讨论那么做是不是一个好主意，
开会讨论那么做是不是一个好主意，
02:49
So here they are having a meeting
to discuss whether it was a good idea,
to discuss whether it was a good idea,
52
157936
3530
片刻后他们下定结论，不是的。
02:53
and after a little while,
they conclude, no,
they conclude, no,
53
161490
3346
那是一个很糟糕的主意。
02:56
this was a terrible idea.
54
164860
1345
我们的物种已经奄奄一息了，
02:58
Our species is in dire straits.
55
166229
1782
03:00
In fact, you can see the existential
sadness in their eyes.
sadness in their eyes.
56
168538
4263
你都可以从它们的眼神中看到这种忧伤，
（笑声）
03:04
(Laughter)
57
172825
1640
所以创造比你自己更聪明的物种，
03:06
So this queasy feeling that making
something smarter than your own species
something smarter than your own species
58
174489
4840
也许不是一个好主意——
03:11
is maybe not a good idea --
59
179353
2365
03:14
what can we do about that?
60
182488
1491
那我们能做些什么呢？
其实没什么能做的，
除了停止研究人工智能，
除了停止研究人工智能，
03:16
Well, really nothing,
except stop doing AI,
except stop doing AI,
61
184003
4767
03:20
and because of all
the benefits that I mentioned
the benefits that I mentioned
62
188794
2510
但因为人工智能能带来
我之前所说的诸多益处，
我之前所说的诸多益处，
也因为我是
人工智能的研究者之一，
人工智能的研究者之一，
03:23
and because I'm an AI researcher,
63
191328
1716
我可不同意就这么止步。
03:25
I'm not having that.
64
193068
1791
03:27
I actually want to be able
to keep doing AI.
to keep doing AI.
65
195283
2468
实际上，我想继续做人工智能。
03:30
So we actually need to nail down
the problem a bit more.
the problem a bit more.
66
198615
2678
所以我们需要把这个问题更细化一点，
它到底是什么呢？
03:33
What exactly is the problem?
67
201317
1371
那就是为什么更强大的
人工智能可能会是灾难呢？
人工智能可能会是灾难呢？
03:34
Why is better AI possibly a catastrophe?
68
202712
3246
03:39
So here's another quotation:
69
207398
1498
再来看这段话：
03:41
"We had better be quite sure
that the purpose put into the machine
that the purpose put into the machine
70
209935
3335
”我们一定得确保我们
给机器输入的目的和价值
给机器输入的目的和价值
是我们确实想要的目的和价值。“
03:45
is the purpose which we really desire."
71
213294
2298
03:48
This was said by Norbert Wiener in 1960,
72
216282
3498
这是诺博特维纳在1960年说的，
他说这话时是刚看到
一个早期的学习系统，
一个早期的学习系统，
03:51
shortly after he watched
one of the very early learning systems
one of the very early learning systems
73
219804
4002
这个系统在学习如何能把
西洋棋下得比它的创造者更好。
西洋棋下得比它的创造者更好。
03:55
learn to play checkers
better than its creator.
better than its creator.
74
223830
2583
04:00
But this could equally have been said
75
228602
2683
与此如出一辙的一句话，
迈达斯国王也说过。
04:03
by King Midas.
76
231309
1167
04:05
King Midas said, "I want everything
I touch to turn to gold,"
I touch to turn to gold,"
77
233083
3134
迈达斯国王说：”我希望
我触碰的所有东西都变成金子。“
我触碰的所有东西都变成金子。“
结果他真的获得了点石成金的能力。
04:08
and he got exactly what he asked for.
78
236241
2473
那就是他所输入的目的，
04:10
That was the purpose
that he put into the machine,
that he put into the machine,
79
238738
2751
从一定程度上说，
04:13
so to speak,
80
241513
1450
后来他的食物、
他的家人都变成了金子，
他的家人都变成了金子，
04:14
and then his food and his drink
and his relatives turned to gold
and his relatives turned to gold
81
242987
3444
他死在痛苦与饥饿之中。
04:18
and he died in misery and starvation.
82
246455
2281
04:22
So we'll call this
"the King Midas problem"
"the King Midas problem"
83
250444
2341
我们可以把这个问题
叫做”迈达斯问题“，
叫做”迈达斯问题“，
这个问题是我们阐述的目标，但实际上
04:24
of stating an objective
which is not, in fact,
which is not, in fact,
84
252809
3305
与我们真正想要的不一致，
04:28
truly aligned with what we want.
85
256138
2413
用现代的术语来说，
我们把它称为”价值一致性问题“。
我们把它称为”价值一致性问题“。
04:30
In modern terms, we call this
"the value alignment problem."
"the value alignment problem."
86
258575
3253
04:37
Putting in the wrong objective
is not the only part of the problem.
is not the only part of the problem.
87
265047
3485
而输入错误的目标
仅仅是问题的一部分。
仅仅是问题的一部分。
它还有另一部分。
04:40
There's another part.
88
268556
1152
04:42
If you put an objective into a machine,
89
270160
1943
如果你为机器输入一个目标，
即便是一个很简单的目标，
比如说”去把咖啡端来“，
比如说”去把咖啡端来“，
04:44
even something as simple as,
"Fetch the coffee,"
"Fetch the coffee,"
90
272127
2448
04:47
the machine says to itself,
91
275908
1841
机器会对自己说：
04:50
"Well, how might I fail
to fetch the coffee?
to fetch the coffee?
92
278733
2623
”好吧，那我要怎么去拿咖啡呢？
说不定有人会把我的电源关掉。
04:53
Someone might switch me off.
93
281380
1580
04:55
OK, I have to take steps to prevent that.
94
283645
2387
好吧，那我要想办法
阻止别人把我关掉。
阻止别人把我关掉。
我得让我的‘关闭’开关失效。
04:58
I will disable my 'off' switch.
95
286056
1906
05:00
I will do anything to defend myself
against interference
against interference
96
288534
2959
我得尽一切可能自我防御，
不让别人干涉我，
不让别人干涉我，
这都是因为我被赋予的目标。”
05:03
with this objective
that I have been given."
that I have been given."
97
291517
2629
这种一根筋的思维，
05:06
So this single-minded pursuit
98
294170
2012
05:09
in a very defensive mode
of an objective that is, in fact,
of an objective that is, in fact,
99
297213
2945
以一种十分防御型的
模式去实现某一目标，
模式去实现某一目标，
实际上与我们人类最初
想实现的目标并不一致——
想实现的目标并不一致——
05:12
not aligned with the true objectives
of the human race --
of the human race --
100
300182
2814
这就是我们面临的问题。
05:16
that's the problem that we face.
101
304122
1862
05:19
And in fact, that's the high-value
takeaway from this talk.
takeaway from this talk.
102
307007
4767
实际上，这就是今天这个演讲的核心。
如果你在我的演讲中只记住一件事，
05:23
If you want to remember one thing,
103
311798
2055
那就是：如果你死了，
你就不能去端咖啡了。
你就不能去端咖啡了。
05:25
it's that you can't fetch
the coffee if you're dead.
the coffee if you're dead.
104
313877
2675
（笑声）
05:28
(Laughter)
105
316576
1061
这很简单。记住它就行了。
每天对自己重复三遍。
每天对自己重复三遍。
05:29
It's very simple. Just remember that.
Repeat it to yourself three times a day.
Repeat it to yourself three times a day.
106
317661
3829
（笑声）
05:33
(Laughter)
107
321514
1821
实际上，这正是电影
05:35
And in fact, this is exactly the plot
108
323359
2754
《2001太空漫步》的剧情。
05:38
of "2001: [A Space Odyssey]"
109
326137
2648
05:41
HAL has an objective, a mission,
110
329226
2090
HAL有一个目标，一个任务，
但这个目标和人类的目标不一致，
05:43
which is not aligned
with the objectives of the humans,
with the objectives of the humans,
111
331340
3732
这就导致了矛盾的产生。
05:47
and that leads to this conflict.
112
335096
1810
05:49
Now fortunately, HAL
is not superintelligent.
is not superintelligent.
113
337494
2969
幸运的是，HAL并不具备超级智能，
他挺聪明的，但还是
比不过人类主角戴夫，
比不过人类主角戴夫，
05:52
He's pretty smart,
but eventually Dave outwits him
but eventually Dave outwits him
114
340487
3587
戴夫成功地把HAL关掉了。
05:56
and manages to switch him off.
115
344098
1849
但我们可能就没有这么幸运了。
06:01
But we might not be so lucky.
116
349828
1619
06:08
So what are we going to do?
117
356193
1592
那我们应该怎么办呢？
06:12
I'm trying to redefine AI
118
360371
2601
我想要重新定义人工智能，
远离传统的定义，
06:14
to get away from this classical notion
119
362996
2061
将其仅限定为
机器通过智能去达成目标。
机器通过智能去达成目标。
06:17
of machines that intelligently
pursue objectives.
pursue objectives.
120
365081
4567
06:22
There are three principles involved.
121
370712
1798
新的定义涉及到三个原则：
第一个原则是利他主义原则，
06:24
The first one is a principle
of altruism, if you like,
of altruism, if you like,
122
372534
3289
也就是说，机器的唯一目标
06:27
that the robot's only objective
123
375847
3262
就是去最大化地实现人类的目标，
06:31
is to maximize the realization
of human objectives,
of human objectives,
124
379133
4246
人类的价值。
06:35
of human values.
125
383403
1390
至于价值，我指的不是感情化的价值，
06:36
And by values here I don't mean
touchy-feely, goody-goody values.
touchy-feely, goody-goody values.
126
384817
3330
而是指人类对生活所向往的，
06:40
I just mean whatever it is
that the human would prefer
that the human would prefer
127
388171
3787
无论是什么。
06:43
their life to be like.
128
391982
1343
06:47
And so this actually violates Asimov's law
129
395364
2309
这实际上违背了阿西莫夫定律，
他指出机器人一定要维护自己的生存。
06:49
that the robot has to protect
its own existence.
its own existence.
130
397697
2329
但我定义的机器
对维护自身生存毫无兴趣。
对维护自身生存毫无兴趣。
06:52
It has no interest in preserving
its existence whatsoever.
its existence whatsoever.
131
400050
3723
06:57
The second law is a law
of humility, if you like.
of humility, if you like.
132
405420
3768
第二个原则不妨称之为谦逊原则。
07:01
And this turns out to be really
important to make robots safe.
important to make robots safe.
133
409974
3743
这一条对于制造安全的机器十分重要。
它说的是机器不知道
07:05
It says that the robot does not know
134
413741
3142
人类的价值是什么，
07:08
what those human values are,
135
416907
2028
机器知道它需要将人类的价值最大化，
却不知道这价值究竟是什么。
却不知道这价值究竟是什么。
07:10
so it has to maximize them,
but it doesn't know what they are.
but it doesn't know what they are.
136
418959
3178
07:15
And that avoids this problem
of single-minded pursuit
of single-minded pursuit
137
423254
2626
为了避免一根筋地追求
某一目标，
07:17
of an objective.
138
425904
1212
这种不确定性是至关重要的。
07:19
This uncertainty turns out to be crucial.
139
427140
2172
07:21
Now, in order to be useful to us,
140
429726
1639
那机器为了对我们有用，
它就得掌握一些
关于我们想要什么的信息。
关于我们想要什么的信息。
07:23
it has to have some idea of what we want.
141
431389
2731
07:27
It obtains that information primarily
by observation of human choices,
by observation of human choices,
142
435223
5427
它主要通过观察人类
做的选择来获取这样的信息，
做的选择来获取这样的信息，
我们自己做出的选择会包含着
07:32
so our own choices reveal information
143
440674
2801
关于我们希望我们的生活
是什么样的信息，
是什么样的信息，
07:35
about what it is that we prefer
our lives to be like.
our lives to be like.
144
443499
3300
07:40
So those are the three principles.
145
448632
1683
这就是三条原则。
让我们来看看它们是如何应用到
07:42
Let's see how that applies
to this question of:
to this question of:
146
450339
2318
像图灵说的那样，
“将机器关掉”这个问题上来。
“将机器关掉”这个问题上来。
07:44
"Can you switch the machine off?"
as Turing suggested.
as Turing suggested.
147
452681
2789
07:49
So here's a PR2 robot.
148
457073
2120
这是一个PR2机器人。
我们实验室里有一个。
07:51
This is one that we have in our lab,
149
459217
1821
它的背面有一个大大的红色的开关。
07:53
and it has a big red "off" switch
right on the back.
right on the back.
150
461062
2903
07:56
The question is: Is it
going to let you switch it off?
going to let you switch it off?
151
464541
2615
那问题来了：它会让你把它关掉吗？
如果我们按传统的方法，
07:59
If we do it the classical way,
152
467180
1465
给它一个目标，让它拿咖啡，
它会想：”我必须去拿咖啡，
它会想：”我必须去拿咖啡，
08:00
we give it the objective of, "Fetch
the coffee, I must fetch the coffee,
the coffee, I must fetch the coffee,
153
468669
3482
但我死了就不能拿咖啡了。“
08:04
I can't fetch the coffee if I'm dead,"
154
472175
2580
显然PR2听过我的演讲了，
08:06
so obviously the PR2
has been listening to my talk,
has been listening to my talk,
155
474779
3341
所以它说：”我必须让我的开关失灵，
08:10
and so it says, therefore,
"I must disable my 'off' switch,
"I must disable my 'off' switch,
156
478144
3753
08:14
and probably taser all the other
people in Starbucks
people in Starbucks
157
482976
2694
可能还要把那些在星巴克里，
可能干扰我的人都电击一下。“
08:17
who might interfere with me."
158
485694
1560
（笑声）
08:19
(Laughter)
159
487278
2062
这看起来必然会发生，对吗？
08:21
So this seems to be inevitable, right?
160
489364
2153
这种失败看起来是必然的，
08:23
This kind of failure mode
seems to be inevitable,
seems to be inevitable,
161
491541
2398
因为机器人在遵循
一个十分确定的目标。
一个十分确定的目标。
08:25
and it follows from having
a concrete, definite objective.
a concrete, definite objective.
162
493963
3543
08:30
So what happens if the machine
is uncertain about the objective?
is uncertain about the objective?
163
498812
3144
那如果机器对目标
不那么确定会发生什么呢？
不那么确定会发生什么呢？
那它的思路就不一样了。
08:33
Well, it reasons in a different way.
164
501980
2127
它会说：”好的，人类可能会把我关掉，
08:36
It says, "OK, the human
might switch me off,
might switch me off,
165
504131
2424
08:39
but only if I'm doing something wrong.
166
507144
1866
但只在我做错事的时候。
08:41
Well, I don't really know what wrong is,
167
509747
2475
我不知道什么是错事，
但我知道我不该做那些事。”
08:44
but I know that I don't want to do it."
168
512246
2044
这就是第一和第二原则。
08:46
So that's the first and second
principles right there.
principles right there.
169
514314
3010
“那我就应该让人类把我关掉。”
08:49
"So I should let the human switch me off."
170
517348
3359
08:53
And in fact you can calculate
the incentive that the robot has
the incentive that the robot has
171
521721
3956
事实上你可以计算出机器人
让人类把它关掉的动机，
08:57
to allow the human to switch it off,
172
525701
2493
而且这个动机是
09:00
and it's directly tied to the degree
173
528218
1914
与对目标的不确定程度直接相关的。
09:02
of uncertainty about
the underlying objective.
the underlying objective.
174
530156
2746
09:05
And then when the machine is switched off,
175
533977
2949
当机器被关闭后，
第三条原则就起作用了。
09:08
that third principle comes into play.
176
536950
1805
机器开始学习它所追求的目标，
09:10
It learns something about the objectives
it should be pursuing,
it should be pursuing,
177
538779
3062
因为它知道它刚做的事是不对的。
09:13
because it learns that
what it did wasn't right.
what it did wasn't right.
178
541865
2533
实际上，我们可以用希腊字母
09:16
In fact, we can, with suitable use
of Greek symbols,
of Greek symbols,
179
544422
3570
就像数学家们经常做的那样，
09:20
as mathematicians usually do,
180
548016
2131
直接证明这一定理，
09:22
we can actually prove a theorem
181
550171
1984
那就是这样的一个机器人
对人们是绝对有利的。
对人们是绝对有利的。
09:24
that says that such a robot
is provably beneficial to the human.
is provably beneficial to the human.
182
552179
3553
可以证明我们的生活
有如此设计的机器人会变得
有如此设计的机器人会变得
09:27
You are provably better off
with a machine that's designed in this way
with a machine that's designed in this way
183
555756
3803
比没有这样的机器人更好。
09:31
than without it.
184
559583
1246
09:33
So this is a very simple example,
but this is the first step
but this is the first step
185
561237
2906
这是一个很简单的例子，但这只是
我们尝试实现与人类
兼容的人工智能的第一步。
兼容的人工智能的第一步。
09:36
in what we're trying to do
with human-compatible AI.
with human-compatible AI.
186
564167
3903
09:42
Now, this third principle,
187
570657
3257
现在来看第三个原则。
我知道你们可能正在
为这一个原则而大伤脑筋。
为这一个原则而大伤脑筋。
09:45
I think is the one that you're probably
scratching your head over.
scratching your head over.
188
573938
3112
你可能会想：“你知道，
我有时不按规矩办事。
我有时不按规矩办事。
09:49
You're probably thinking, "Well,
you know, I behave badly.
you know, I behave badly.
189
577074
3239
我可不希望我的机器人
像我一样行事。
像我一样行事。
09:52
I don't want my robot to behave like me.
190
580337
2929
我有时大半夜偷偷摸摸地
从冰箱里找东西吃，
从冰箱里找东西吃，
09:55
I sneak down in the middle of the night
and take stuff from the fridge.
and take stuff from the fridge.
191
583290
3434
诸如此类的事。”
09:58
I do this and that."
192
586748
1168
有各种各样的事你是
不希望机器人去做的。
不希望机器人去做的。
09:59
There's all kinds of things
you don't want the robot doing.
you don't want the robot doing.
193
587940
2797
但实际上并不一定会这样。
10:02
But in fact, it doesn't
quite work that way.
quite work that way.
194
590761
2071
仅仅是因为你表现不好，
10:04
Just because you behave badly
195
592856
2155
并不代表机器人就会复制你的行为。
10:07
doesn't mean the robot
is going to copy your behavior.
is going to copy your behavior.
196
595035
2623
它会去尝试理解你做事的动机，
而且可能会在合适的情况下制止你去做
而且可能会在合适的情况下制止你去做
10:09
It's going to understand your motivations
and maybe help you resist them,
and maybe help you resist them,
197
597682
3910
那些不该做的事。
10:13
if appropriate.
198
601616
1320
10:16
But it's still difficult.
199
604206
1464
但这仍然十分困难。
10:18
What we're trying to do, in fact,
200
606302
2545
实际上，我们在做的是
让机器去预测任何一个人，
在他们的任何一种
在他们的任何一种
10:20
is to allow machines to predict
for any person and for any possible life
for any person and for any possible life
201
608871
5796
可能的生活中
10:26
that they could live,
202
614691
1161
以及别人的生活中，
10:27
and the lives of everybody else:
203
615876
1597
10:29
Which would they prefer?
204
617497
2517
他们会更倾向于哪一种？
10:34
And there are many, many
difficulties involved in doing this;
difficulties involved in doing this;
205
622061
2954
这涉及到诸多困难；
我不认为这会很快地就被解决。
10:37
I don't expect that this
is going to get solved very quickly.
is going to get solved very quickly.
206
625039
2932
实际上，真正的困难是我们自己。
10:39
The real difficulties, in fact, are us.
207
627995
2643
10:44
As I have already mentioned,
we behave badly.
we behave badly.
208
632149
3117
就像我刚说的那样，
我们做事不守规矩，
我们做事不守规矩，
我们中有的人甚至行为肮脏。
10:47
In fact, some of us are downright nasty.
209
635290
2321
10:50
Now the robot, as I said,
doesn't have to copy the behavior.
doesn't have to copy the behavior.
210
638431
3052
就像我说的，
机器人并不会复制那些行为，
机器人并不会复制那些行为，
机器人没有自己的目标，
10:53
The robot does not have
any objective of its own.
any objective of its own.
211
641507
2791
它是完全无私的。
10:56
It's purely altruistic.
212
644322
1737
10:59
And it's not designed just to satisfy
the desires of one person, the user,
the desires of one person, the user,
213
647293
5221
它的设计不是去满足
某一个人、一个用户的欲望，
某一个人、一个用户的欲望，
而是去尊重所有人的意愿。
11:04
but in fact it has to respect
the preferences of everybody.
the preferences of everybody.
214
652538
3138
11:09
So it can deal with a certain
amount of nastiness,
amount of nastiness,
215
657263
2570
所以它能对付一定程度的肮脏行为。
它甚至能理解你的不端行为，比如说
11:11
and it can even understand
that your nastiness, for example,
that your nastiness, for example,
216
659857
3701
假如你是一个边境护照官员，
很可能收取贿赂，
很可能收取贿赂，
11:15
you may take bribes as a passport official
217
663582
2671
因为你得养家、
得供你的孩子们上学。
得供你的孩子们上学。
11:18
because you need to feed your family
and send your kids to school.
and send your kids to school.
218
666277
3812
机器人能理解这一点，
它不会因此去偷，
它不会因此去偷，
11:22
It can understand that;
it doesn't mean it's going to steal.
it doesn't mean it's going to steal.
219
670113
2906
它反而会帮助你去供孩子们上学。
11:25
In fact, it'll just help you
send your kids to school.
send your kids to school.
220
673043
2679
11:28
We are also computationally limited.
221
676976
3012
我们的计算能力也是有限的。
李世石是一个杰出的围棋大师，
11:32
Lee Sedol is a brilliant Go player,
222
680012
2505
但他还是输了。
11:34
but he still lost.
223
682541
1325
如果我们看他的行动，
他最终输掉了棋局。
他最终输掉了棋局。
11:35
So if we look at his actions,
he took an action that lost the game.
he took an action that lost the game.
224
683890
4239
但这不意味着他想要输。
11:40
That doesn't mean he wanted to lose.
225
688153
2161
所以要理解他的行为，
11:43
So to understand his behavior,
226
691340
2040
我们得从人类认知模型来反过来想，
11:45
we actually have to invert
through a model of human cognition
through a model of human cognition
227
693404
3644
这包含了我们的计算能力限制，
是一个很复杂的模型，
是一个很复杂的模型，
11:49
that includes our computational
limitations -- a very complicated model.
limitations -- a very complicated model.
228
697072
4977
但仍然是我们可以尝试去理解的。
11:54
But it's still something
that we can work on understanding.
that we can work on understanding.
229
702073
2993
11:57
Probably the most difficult part,
from my point of view as an AI researcher,
from my point of view as an AI researcher,
230
705876
4320
可能对于我这样一个
人工智能研究人员来说最大的困难，
人工智能研究人员来说最大的困难，
是我们彼此各不相同。
12:02
is the fact that there are lots of us,
231
710220
2575
12:06
and so the machine has to somehow
trade off, weigh up the preferences
trade off, weigh up the preferences
232
714294
3581
所以机器必须想办法去判别衡量
不同人的不同需求，
12:09
of many different people,
233
717899
2225
而又有众多方法去做这样的判断。
12:12
and there are different ways to do that.
234
720148
1906
经济学家、社会学家、
哲学家都理解这一点，
哲学家都理解这一点，
12:14
Economists, sociologists,
moral philosophers have understood that,
moral philosophers have understood that,
235
722078
3689
我们正在积极地去寻求合作。
12:17
and we are actively
looking for collaboration.
looking for collaboration.
236
725791
2455
让我们来看看如果我们
把这一步弄错了会怎么样。
把这一步弄错了会怎么样。
12:20
Let's have a look and see what happens
when you get that wrong.
when you get that wrong.
237
728270
3251
举例来说，你可能会
与你的人工智能助理，
与你的人工智能助理，
12:23
So you can have
a conversation, for example,
a conversation, for example,
238
731545
2133
有这样的对话：
12:25
with your intelligent personal assistant
239
733702
1944
这样的人工智能可能几年内就会出现，
12:27
that might be available
in a few years' time.
in a few years' time.
240
735670
2285
可以把它想做加强版的Siri。
12:29
Think of a Siri on steroids.
241
737979
2524
12:33
So Siri says, "Your wife called
to remind you about dinner tonight."
to remind you about dinner tonight."
242
741627
4322
Siri对你说：“你的妻子打电话
提醒你今晚要跟她共进晚餐。”
提醒你今晚要跟她共进晚餐。”
12:38
And of course, you've forgotten.
"What? What dinner?
"What? What dinner?
243
746616
2508
而你呢，自然忘了这回事：
“什么？什么晚饭？
“什么？什么晚饭？
你在说什么？”
12:41
What are you talking about?"
244
749148
1425
12:42
"Uh, your 20th anniversary at 7pm."
245
750597
3746
“啊，你们晚上7点，
庆祝结婚20周年纪念日。”
庆祝结婚20周年纪念日。”
12:48
"I can't do that. I'm meeting
with the secretary-general at 7:30.
with the secretary-general at 7:30.
246
756915
3719
“我可去不了。
我约了晚上7点半见领导。
我约了晚上7点半见领导。
怎么会这样呢？”
12:52
How could this have happened?"
247
760658
1692
“呃，我可是提醒过你的，
但你不听我的建议。”
但你不听我的建议。”
12:54
"Well, I did warn you, but you overrode
my recommendation."
my recommendation."
248
762374
4660
13:00
"Well, what am I going to do?
I can't just tell him I'm too busy."
I can't just tell him I'm too busy."
249
768146
3328
“我该怎么办呢？我可不能
跟领导说我有事，没空见他。”
跟领导说我有事，没空见他。”
13:04
"Don't worry. I arranged
for his plane to be delayed."
for his plane to be delayed."
250
772490
3281
“别担心。我已经安排了，
让他的航班延误。
让他的航班延误。
（笑声）
13:07
(Laughter)
251
775795
1682
13:10
"Some kind of computer malfunction."
252
778249
2101
“像是因为某种计算机故障那样。”
（笑声）
13:12
(Laughter)
253
780374
1212
“真的吗？这个你也能做到？”
13:13
"Really? You can do that?"
254
781610
1617
13:16
"He sends his profound apologies
255
784400
2179
“领导很不好意思，跟你道歉，
并且告诉你明天
中午午饭不见不散。”
中午午饭不见不散。”
13:18
and looks forward to meeting you
for lunch tomorrow."
for lunch tomorrow."
256
786603
2555
（笑声）
13:21
(Laughter)
257
789182
1299
这里就有一个小小的问题。
13:22
So the values here --
there's a slight mistake going on.
there's a slight mistake going on.
258
790505
4403
这显然是在遵循我妻子的价值论，
13:26
This is clearly following my wife's values
259
794932
3009
那就是“老婆开心，生活舒心”。
13:29
which is "Happy wife, happy life."
260
797965
2069
（笑声）
13:32
(Laughter)
261
800058
1583
它也有可能发展成另一种情况。
13:33
It could go the other way.
262
801665
1444
13:35
You could come home
after a hard day's work,
after a hard day's work,
263
803821
2201
你忙碌一天，回到家里，
电脑对你说：“像是繁忙的一天啊？”
13:38
and the computer says, "Long day?"
264
806046
2195
“是啊，我连午饭都没来得及吃。”
13:40
"Yes, I didn't even have time for lunch."
265
808265
2288
“那你一定很饿了吧。”
13:42
"You must be very hungry."
266
810577
1282
“快饿晕了。你能做点晚饭吗？”
13:43
"Starving, yeah.
Could you make some dinner?"
Could you make some dinner?"
267
811883
2646
13:48
"There's something I need to tell you."
268
816070
2090
“有一件事我得告诉你。
（笑声）
13:50
(Laughter)
269
818184
1155
13:52
"There are humans in South Sudan
who are in more urgent need than you."
who are in more urgent need than you."
270
820193
4905
”南苏丹的人们可比你更需要照顾。
（笑声）
13:57
(Laughter)
271
825122
1104
“所以我要离开了。
你自己做饭去吧。”
你自己做饭去吧。”
13:58
"So I'm leaving. Make your own dinner."
272
826250
2075
（笑声）
14:00
(Laughter)
273
828349
2000
我们得解决这些问题，
14:02
So we have to solve these problems,
274
830823
1739
我也很期待去解决。
14:04
and I'm looking forward
to working on them.
to working on them.
275
832586
2515
我们有理由感到乐观。
14:07
There are reasons for optimism.
276
835125
1843
理由之一是
14:08
One reason is,
277
836992
1159
我们有大量的数据，
14:10
there is a massive amount of data.
278
838175
1868
记住，我说过机器将能够阅读一切
14:12
Because remember -- I said
they're going to read everything
they're going to read everything
279
840067
2794
人类所写下来的东西，
14:14
the human race has ever written.
280
842885
1546
而我们写下的大多数是
我们做的什么事情，
我们做的什么事情，
14:16
Most of what we write about
is human beings doing things
is human beings doing things
281
844455
2724
以及其他人对此有什么意见。
14:19
and other people getting upset about it.
282
847203
1914
所以机器可以从大量的数据中去学习。
14:21
So there's a massive amount
of data to learn from.
of data to learn from.
283
849141
2398
同时从经济的角度，
我们也有足够的动机
我们也有足够的动机
14:23
There's also a very
strong economic incentive
strong economic incentive
284
851563
2236
14:27
to get this right.
285
855331
1186
去把这件事做对。
想象一下，你家里有个居家机器人，
14:28
So imagine your domestic robot's at home.
286
856541
2001
而你又得加班，
机器人得给孩子们做饭,
机器人得给孩子们做饭,
14:30
You're late from work again
and the robot has to feed the kids,
and the robot has to feed the kids,
287
858566
3067
孩子们很饿，
但冰箱里什么都没有。
但冰箱里什么都没有。
14:33
and the kids are hungry
and there's nothing in the fridge.
and there's nothing in the fridge.
288
861657
2823
然后机器人看到了家里的猫，
14:36
And the robot sees the cat.
289
864504
2605
（笑声）
14:39
(Laughter)
290
867133
1692
机器人还没学透人类的价值论，
14:40
And the robot hasn't quite learned
the human value function properly,
the human value function properly,
291
868849
4190
所以它不知道
14:45
so it doesn't understand
292
873063
1251
猫的感情价值
大于猫的营养价值。
大于猫的营养价值。
14:46
the sentimental value of the cat outweighs
the nutritional value of the cat.
the nutritional value of the cat.
293
874338
4844
（笑声）
14:51
(Laughter)
294
879206
1095
接下来会发生什么？
14:52
So then what happens?
295
880325
1748
差不多是这样的：
14:54
Well, it happens like this:
296
882097
3297
头版头条：“疯狂的机器人
把猫煮了给主人当晚饭！”
把猫煮了给主人当晚饭！”
14:57
"Deranged robot cooks kitty
for family dinner."
for family dinner."
297
885418
2964
这一个事故就足以结束
整个居家机器人产业。
整个居家机器人产业。
15:00
That one incident would be the end
of the domestic robot industry.
of the domestic robot industry.
298
888406
4523
所以我们有足够的动机在我们实现
15:04
So there's a huge incentive
to get this right
to get this right
299
892953
3372
超级智能机器让它更加完善。
15:08
long before we reach
superintelligent machines.
superintelligent machines.
300
896349
2715
15:12
So to summarize:
301
900128
1535
总结来说：
我想要改变人工智能的定义，
15:13
I'm actually trying to change
the definition of AI
the definition of AI
302
901687
2881
让我们可以证明机器对我们是有利的。
15:16
so that we have provably
beneficial machines.
beneficial machines.
303
904592
2993
这三个原则是：
15:19
And the principles are:
304
907609
1222
机器是利他的，
15:20
machines that are altruistic,
305
908855
1398
只想着实现我们的目标，
15:22
that want to achieve only our objectives,
306
910277
2804
但它不确定我们的目标是什么，
15:25
but that are uncertain
about what those objectives are,
about what those objectives are,
307
913105
3116
所以它会观察我们，
15:28
and will watch all of us
308
916245
1998
从中学习我们想要的究竟是什么。
15:30
to learn more about what it is
that we really want.
that we really want.
309
918267
3203
15:34
And hopefully in the process,
we will learn to be better people.
we will learn to be better people.
310
922373
3559
希望在这个过程中，
我们也能学会成为更好的人。
我们也能学会成为更好的人。
谢谢大家。
15:37
Thank you very much.
311
925956
1191
（掌声）
15:39
(Applause)
312
927171
3709
克里斯安德森：
非常有意思，斯图尔特。
非常有意思，斯图尔特。
15:42
Chris Anderson: So interesting, Stuart.
313
930904
1868
我们趁着工作人员
为下一位演讲者布置的时候
为下一位演讲者布置的时候
15:44
We're going to stand here a bit
because I think they're setting up
because I think they're setting up
314
932796
3170
来简单聊几句。
15:47
for our next speaker.
315
935990
1151
15:49
A couple of questions.
316
937165
1538
我有几个问题。
从直觉上来看，将无知编入到程序中
似乎是一个很重要的理念，
似乎是一个很重要的理念，
15:50
So the idea of programming in ignorance
seems intuitively really powerful.
seems intuitively really powerful.
317
938727
5453
当你要实现超级智能时，
15:56
As you get to superintelligence,
318
944204
1594
什么能阻止机器人？
15:57
what's going to stop a robot
319
945822
2258
当它在阅读和学习的过程中发现，
16:00
reading literature and discovering
this idea that knowledge
this idea that knowledge
320
948104
2852
16:02
is actually better than ignorance
321
950980
1572
知识比无知更强大，
16:04
and still just shifting its own goals
and rewriting that programming?
and rewriting that programming?
322
952576
4218
然后就改变它的目标
去重新编写程序呢？
去重新编写程序呢？
斯图尔特拉塞尔：是的，
我们想要它去学习，就像我说的，
我们想要它去学习，就像我说的，
16:09
Stuart Russell: Yes, so we want
it to learn more, as I said,
it to learn more, as I said,
323
957692
6356
学习我们的目标。
16:16
about our objectives.
324
964072
1287
16:17
It'll only become more certain
as it becomes more correct,
as it becomes more correct,
325
965383
5521
它只有在理解得越来越正确的时候，
才会变得更确定，
才会变得更确定，
16:22
so the evidence is there
326
970928
1945
我们有证据显示，
它的设计使它能按正确的方式理解。
16:24
and it's going to be designed
to interpret it correctly.
to interpret it correctly.
327
972897
2724
16:27
It will understand, for example,
that books are very biased
that books are very biased
328
975645
3956
比如说，它能够理解书中的论证是
带有非常强的偏见的。
16:31
in the evidence they contain.
329
979625
1483
书中只会讲述国王、王子
16:33
They only talk about kings and princes
330
981132
2397
和那些精英白人男性做的事。
16:35
and elite white male people doing stuff.
331
983553
2800
这是一个复杂的问题，
16:38
So it's a complicated problem,
332
986377
2096
但当它更深入地学习我们的目标时，
16:40
but as it learns more about our objectives
333
988497
3872
16:44
it will become more and more useful to us.
334
992393
2063
它就变得对我们更有用。
CA：那你不能把这些
都集中在一条准则里吗？
都集中在一条准则里吗？
16:46
CA: And you couldn't
just boil it down to one law,
just boil it down to one law,
335
994480
2526
把这样的命令写在它的程序里：
16:49
you know, hardwired in:
336
997030
1650
“如果人类什么时候想把我关掉，
16:50
"if any human ever tries to switch me off,
337
998704
3293
我服从。我服从。”
16:54
I comply. I comply."
338
1002021
1935
SR：绝对不行，
16:55
SR: Absolutely not.
339
1003980
1182
那将是一个很糟糕的主意。
16:57
That would be a terrible idea.
340
1005186
1499
试想一下，你有一辆无人驾驶汽车，
16:58
So imagine that you have
a self-driving car
a self-driving car
341
1006709
2689
你想让它送你五岁的孩子
17:01
and you want to send your five-year-old
342
1009422
2433
去上学。
17:03
off to preschool.
343
1011879
1174
你希望你五岁的孩子
能在汽车运行过程中
能在汽车运行过程中
17:05
Do you want your five-year-old
to be able to switch off the car
to be able to switch off the car
344
1013077
3101
将它关闭吗？
17:08
while it's driving along?
345
1016202
1213
应该不会吧。
17:09
Probably not.
346
1017439
1159
它得理解下指令的人有多理智，
是不是讲道理。
是不是讲道理。
17:10
So it needs to understand how rational
and sensible the person is.
and sensible the person is.
347
1018622
4703
这个人越理智，
17:15
The more rational the person,
348
1023349
1676
它就越愿意自己被关掉。
17:17
the more willing you are
to be switched off.
to be switched off.
349
1025049
2103
如果这个人是完全思绪混乱
或者甚至是有恶意的，
或者甚至是有恶意的，
17:19
If the person is completely
random or even malicious,
random or even malicious,
350
1027176
2543
那你就不愿意它被关掉。
17:21
then you're less willing
to be switched off.
to be switched off.
351
1029743
2512
CA：好吧。斯图尔特，我得说
17:24
CA: All right. Stuart, can I just say,
352
1032279
1866
我真的希望你为我们
能把这一切研究出来，
能把这一切研究出来，
17:26
I really, really hope you
figure this out for us.
figure this out for us.
353
1034169
2314
很感谢你的演讲，太精彩了。
17:28
Thank you so much for that talk.
That was amazing.
That was amazing.
354
1036507
2375
SR：谢谢。
17:30
SR: Thank you.
355
1038906
1167
（掌声）
17:32
(Applause)
356
1040097
1837
ABOUT THE SPEAKER
Stuart Russell - AI expertStuart Russell wrote the standard text on AI; now he thinks deeply on AI's future -- and the future of us humans, too.
Why you should listen
Stuart Russell is a professor (and formerly chair) of Electrical Engineering and Computer Sciences at University of California at Berkeley. His book Artificial Intelligence: A Modern Approach (with Peter Norvig) is the standard text in AI; it has been translated into 13 languages and is used in more than 1,300 universities in 118 countries. His research covers a wide range of topics in artificial intelligence including machine learning, probabilistic reasoning, knowledge representation, planning, real-time decision making, multitarget tracking, computer vision, computational physiology, global seismic monitoring and philosophical foundations.
He also works for the United Nations, developing a new global seismic monitoring system for the nuclear-test-ban treaty. His current concerns include the threat of autonomous weapons and the long-term future of artificial intelligence and its relation to humanity.
Stuart Russell | Speaker | TED.com