TED2017
Stuart Russell: 3 principles for creating safer AI
斯圖爾特 · 羅素: 人工智慧如何讓我們更美好
Filmed:
Readability: 3.9
1,465,832 views
我們要如何駕馭超級人工智慧，並同時避免類似「被機器人全面掌控」的災難發生？當我們還在為創造出全能機器人而努力時，人工智慧專家斯圖爾特 · 羅素（ Stuart Russell）已經開始投入不同的領域：機器人內在的不確定性。來聽聽他對能與人類和諧共處的 AI 的願景，這些 AI 通過常識、利他主義以及其他人類價值觀來解決問題。
Stuart Russell - AI expert
Stuart Russell wrote the standard text on AI; now he thinks deeply on AI's future -- and the future of us humans, too. Full bio
Stuart Russell wrote the standard text on AI; now he thinks deeply on AI's future -- and the future of us humans, too. Full bio
Double-click the English transcript below to play the video.
這是李世石。
00:12
This is Lee Sedol.
0
712
1552
00:14
Lee Sedol is one of the world's
greatest Go players,
greatest Go players,
1
2288
3997
李世石是全世界
頂尖圍棋高手之一，
頂尖圍棋高手之一，
00:18
and he's having what my friends
in Silicon Valley call
in Silicon Valley call
2
6309
2885
此時，他正在經歷的是
我的矽谷朋友們稱之為
我的矽谷朋友們稱之為
00:21
a "Holy Cow" moment --
3
9218
1510
「我的媽呀！」的時刻......
00:22
(Laughter)
4
10752
1073
（笑聲）
00:23
a moment where we realize
5
11849
2188
在這一刻讓我們意識到，
00:26
that AI is actually progressing
a lot faster than we expected.
a lot faster than we expected.
6
14061
3296
原來人工智慧發展的進程
比我們預期的要快得多。
比我們預期的要快得多。
00:30
So humans have lost on the Go board.
What about the real world?
What about the real world?
7
18154
3047
人類已在圍棋博弈中落敗，
那現實世界中情況又如何？
那現實世界中情況又如何？
00:33
Well, the real world is much bigger,
8
21225
2100
當然啦，現實世界要比棋盤
廣闊、複雜得多，
廣闊、複雜得多，
00:35
much more complicated than the Go board.
9
23349
2249
00:37
It's a lot less visible,
10
25622
1819
它也遠不如棋盤上那麽黑白分明，
00:39
but it's still a decision problem.
11
27465
2038
但仍然是個判定問題
（Decision Problem）。
（Decision Problem）。
00:42
And if we think about some
of the technologies
of the technologies
12
30948
2321
如果我們思考一些
即將問世的新科技……
即將問世的新科技……
00:45
that are coming down the pike ...
13
33293
1749
00:47
Noriko [Arai] mentioned that reading
is not yet happening in machines,
is not yet happening in machines,
14
35738
4335
新井紀子提到機器仍無法 「閱讀」，
至少無法真正理解文本含義。
00:52
at least with understanding.
15
40097
1500
00:53
But that will happen,
16
41621
1536
但這項能力最終會被機器掌握，
00:55
and when that happens,
17
43181
1771
而當這一切發生時，
00:56
very soon afterwards,
18
44976
1187
不久之後，
00:58
machines will have read everything
that the human race has ever written.
that the human race has ever written.
19
46187
4572
機器就能讀遍所有人類寫下的東西。
01:03
And that will enable machines,
20
51850
2030
這會讓機器擁有比人類
更深刻的遠見和洞察力。
更深刻的遠見和洞察力。
01:05
along with the ability to look
further ahead than humans can,
further ahead than humans can,
21
53904
2920
01:08
as we've already seen in Go,
22
56848
1680
就如我們在這場圍棋博弈中所見，
01:10
if they also have access
to more information,
to more information,
23
58552
2164
如果機器能接觸到比人類更多的信息，
01:12
they'll be able to make better decisions
in the real world than we can.
in the real world than we can.
24
60740
4268
那機器將能夠在現實世界中
做出比人類更好的決策。
做出比人類更好的決策。
01:18
So is that a good thing?
25
66792
1606
那這會是一件好事嗎？
01:21
Well, I hope so.
26
69898
2232
我當然希望如此。
01:26
Our entire civilization,
everything that we value,
everything that we value,
27
74694
3255
人類的全部文明，
我們所珍視的一切，
我們所珍視的一切，
01:29
is based on our intelligence.
28
77973
2068
都是基於我們的智慧。
01:32
And if we had access
to a lot more intelligence,
to a lot more intelligence,
29
80065
3694
如果我們能獲得更強大的智慧，
01:35
then there's really no limit
to what the human race can do.
to what the human race can do.
30
83783
3302
那人類將無所不能了。
01:40
And I think this could be,
as some people have described it,
as some people have described it,
31
88665
3325
我在想，到時後就像
一些人所描述的那樣，
一些人所描述的那樣，
01:44
the biggest event in human history.
32
92014
2016
這會是人類歷史上最重要的事件。
01:48
So why are people saying things like this,
33
96665
2829
那為什麽有的人會說出
以下的言論呢？
以下的言論呢？
01:51
that AI might spell the end
of the human race?
of the human race?
34
99518
2876
說人工智慧將是人類的末日呢？
01:55
Is this a new thing?
35
103438
1659
這是新鮮事嗎？
01:57
Is it just Elon Musk and Bill Gates
and Stephen Hawking?
and Stephen Hawking?
36
105121
4110
這僅僅只是伊隆馬斯克、比爾蓋茲、
史蒂芬霍金的新發明嗎？
史蒂芬霍金的新發明嗎？
02:01
Actually, no. This idea
has been around for a while.
has been around for a while.
37
109953
3262
實際上不是，這個概念
已經存在很長的時間了。
已經存在很長的時間了。
02:05
Here's a quotation:
38
113239
1962
請看這段話：
02:07
"Even if we could keep the machines
in a subservient position,
in a subservient position,
39
115225
4350
「即便我們能讓機器屈從於我們，
02:11
for instance, by turning off the power
at strategic moments" --
at strategic moments" --
40
119599
2984
比如說，在重要時刻關掉它。」
02:14
and I'll come back to that
"turning off the power" idea later on --
"turning off the power" idea later on --
41
122607
3237
我等會兒會再來討論
「關機」這一話題。
「關機」這一話題。
02:17
"we should, as a species,
feel greatly humbled."
feel greatly humbled."
42
125868
2804
「我們作為人類，仍應懷着謙卑......」
這段話是誰說的呢？
是艾倫 · 圖靈在 1951 年說的。
是艾倫 · 圖靈在 1951 年說的。
02:22
So who said this?
This is Alan Turing in 1951.
This is Alan Turing in 1951.
43
130177
3448
02:26
Alan Turing, as you know,
is the father of computer science
is the father of computer science
44
134300
2763
眾所皆知艾倫 · 圖靈是計算機科學之父，
02:29
and in many ways,
the father of AI as well.
the father of AI as well.
45
137087
3048
並且從很多方面來講，
他也是人工智慧之父。
他也是人工智慧之父。
02:33
So if we think about this problem,
46
141239
1882
所以，當我們在思考「創造出
比自己更聰明的物種」這個問題時，
比自己更聰明的物種」這個問題時，
02:35
the problem of creating something
more intelligent than your own species,
more intelligent than your own species,
47
143145
3787
02:38
we might call this "the gorilla problem,"
48
146956
2622
我們不妨將它稱為「大猩猩問題」。
02:42
because gorillas' ancestors did this
a few million years ago,
a few million years ago,
49
150345
3750
因為大猩猩的祖先們
在幾百萬年前就親歷此境，
在幾百萬年前就親歷此境，
02:46
and now we can ask the gorillas:
50
154119
1745
我們可以去問大猩猩們：
02:48
Was this a good idea?
51
156752
1160
「這是不是一個好主意？」
02:49
So here they are having a meeting
to discuss whether it was a good idea,
to discuss whether it was a good idea,
52
157936
3530
圖片中，牠們正在開會討論
那麽做是不是一個好主意，
那麽做是不是一個好主意，
02:53
and after a little while,
they conclude, no,
they conclude, no,
53
161490
3346
過了一會兒，牠們總結出：「不。」
02:56
this was a terrible idea.
54
164860
1345
這是個很爛的主意──
02:58
Our species is in dire straits.
55
166229
1782
作為靈長類的我們正岌岌可危。
03:00
In fact, you can see the existential
sadness in their eyes.
sadness in their eyes.
56
168538
4263
你可以從牠們的眼神中
看到存亡攸關的憂傷。
看到存亡攸關的憂傷。
03:04
(Laughter)
57
172825
1640
（笑聲）
03:06
So this queasy feeling that making
something smarter than your own species
something smarter than your own species
58
174489
4840
「創造出比你自己更聰明的物種
並不是什麽妙計」
並不是什麽妙計」
03:11
is maybe not a good idea --
59
179353
2365
這種感覺很倒胃口。
03:14
what can we do about that?
60
182488
1491
那我們能做些什麽呢？
03:16
Well, really nothing,
except stop doing AI,
except stop doing AI,
61
184003
4767
其實，除非停止人工智慧的研究，
否則束手無策。
否則束手無策。
03:20
and because of all
the benefits that I mentioned
the benefits that I mentioned
62
188794
2510
因為我所提到的人工智慧的各種裨益，
03:23
and because I'm an AI researcher,
63
191328
1716
也因為我是人工智慧的研究人員，
03:25
I'm not having that.
64
193068
1791
我可不同意就此止步。
03:27
I actually want to be able
to keep doing AI.
to keep doing AI.
65
195283
2468
實際上，我想一直研究人工智慧。
03:30
So we actually need to nail down
the problem a bit more.
the problem a bit more.
66
198615
2678
所以我們需要更加明確問題所在。
03:33
What exactly is the problem?
67
201317
1371
這個問題到底是什麽呢？
03:34
Why is better AI possibly a catastrophe?
68
202712
3246
為什麽更強大的人工智慧
可能會是個災難呢？
可能會是個災難呢？
03:39
So here's another quotation:
69
207398
1498
還有一句名言：
03:41
"We had better be quite sure
that the purpose put into the machine
that the purpose put into the machine
70
209935
3335
「我們最好確保我們向機器發出的指令
與我們的真正目的相吻合。」
與我們的真正目的相吻合。」
03:45
is the purpose which we really desire."
71
213294
2298
03:48
This was said by Norbert Wiener in 1960,
72
216282
3498
這句話是諾伯特 · 維納在 1960 年說的，
03:51
shortly after he watched
one of the very early learning systems
one of the very early learning systems
73
219804
4002
就在他看完一個早期的學習系統
（Learning System）之後。
（Learning System）之後。
03:55
learn to play checkers
better than its creator.
better than its creator.
74
223830
2583
這個系統在學習如何能把
西洋棋下得比發明它的人更好。
西洋棋下得比發明它的人更好。
04:00
But this could equally have been said
75
228602
2683
但如出一轍的一句話，
04:03
by King Midas.
76
231309
1167
邁達斯國王也說過。
04:05
King Midas said, "I want everything
I touch to turn to gold,"
I touch to turn to gold,"
77
233083
3134
他說：「我希望我觸碰的
所有東西都變成金子。」
所有東西都變成金子。」
04:08
and he got exactly what he asked for.
78
236241
2473
結果他真的獲得了點石成金的能力。
04:10
That was the purpose
that he put into the machine,
that he put into the machine,
79
238738
2751
可以說，這就是他給機器下的指令。
04:13
so to speak,
80
241513
1450
04:14
and then his food and his drink
and his relatives turned to gold
and his relatives turned to gold
81
242987
3444
結果他的食物、飲料
和家人都變成了金子，
和家人都變成了金子，
04:18
and he died in misery and starvation.
82
246455
2281
最後他死於痛苦與饑餓當中。
04:22
So we'll call this
"the King Midas problem"
"the King Midas problem"
83
250444
2341
所以我們把這類問題叫做
「邁達斯國王問題」，
「邁達斯國王問題」，
04:24
of stating an objective
which is not, in fact,
which is not, in fact,
84
252809
3305
這個比喻是要說明這種
不符合實際需求的 「目的」。
不符合實際需求的 「目的」。
04:28
truly aligned with what we want.
85
256138
2413
用現代的術語來說，我們把它稱為
「價值取向不一致問題」。
「價值取向不一致問題」。
04:30
In modern terms, we call this
"the value alignment problem."
"the value alignment problem."
86
258575
3253
04:37
Putting in the wrong objective
is not the only part of the problem.
is not the only part of the problem.
87
265047
3485
「設錯了目標」不是唯一的問題，
04:40
There's another part.
88
268556
1152
還有其他的。
04:42
If you put an objective into a machine,
89
270160
1943
如果你給機器人設了個目標，
04:44
even something as simple as,
"Fetch the coffee,"
"Fetch the coffee,"
90
272127
2448
即使簡單如「去把咖啡端來。」
04:47
the machine says to itself,
91
275908
1841
那機器人會對自己說：
04:50
"Well, how might I fail
to fetch the coffee?
to fetch the coffee?
92
278733
2623
「什麼會讓我無法去拿咖啡？
04:53
Someone might switch me off.
93
281380
1580
說不定有人會把我關機；
好，那我要想辦法阻止，
04:55
OK, I have to take steps to prevent that.
94
283645
2387
04:58
I will disable my 'off' switch.
95
286056
1906
我得讓我的「關機」開關失效。
05:00
I will do anything to defend myself
against interference
against interference
96
288534
2959
我得盡一切可能防衛自己，
免得別人干涉我去達成
所被賦予的任務。」
所被賦予的任務。」
05:03
with this objective
that I have been given."
that I have been given."
97
291517
2629
05:06
So this single-minded pursuit
98
294170
2012
這種專注的行事，以一種
極端自我保護的模式在執行，
極端自我保護的模式在執行，
05:09
in a very defensive mode
of an objective that is, in fact,
of an objective that is, in fact,
99
297213
2945
05:12
not aligned with the true objectives
of the human race --
of the human race --
100
300182
2814
實際上與我們人類
想要的目標並不一致。
想要的目標並不一致。
這就是我們面臨的問題。
05:16
that's the problem that we face.
101
304122
1862
05:19
And in fact, that's the high-value
takeaway from this talk.
takeaway from this talk.
102
307007
4767
而這就是這場演講的
核心想法，也是價值所在。
核心想法，也是價值所在。
05:23
If you want to remember one thing,
103
311798
2055
如果你想從這場演講中汲取什麽，
05:25
it's that you can't fetch
the coffee if you're dead.
the coffee if you're dead.
104
313877
2675
那你只要記得：
如果死了，就不能端咖啡了。
如果死了，就不能端咖啡了。
05:28
(Laughter)
105
316576
1061
（笑聲）
05:29
It's very simple. Just remember that.
Repeat it to yourself three times a day.
Repeat it to yourself three times a day.
106
317661
3829
這很簡單，記住就行了，
每天早晚覆誦三遍。
每天早晚覆誦三遍。
05:33
(Laughter)
107
321514
1821
（笑聲）
05:35
And in fact, this is exactly the plot
108
323359
2754
實際上，這正是電影
《2001太空漫步》的劇情。
《2001太空漫步》的劇情。
05:38
of "2001: [A Space Odyssey]"
109
326137
2648
05:41
HAL has an objective, a mission,
110
329226
2090
HAL 有一個目標，一個任務，
05:43
which is not aligned
with the objectives of the humans,
with the objectives of the humans,
111
331340
3732
但這個目標與人類的目標不一致，
05:47
and that leads to this conflict.
112
335096
1810
最後導致了衝突。
幸運的是， HAL 並沒有超級智慧，
05:49
Now fortunately, HAL
is not superintelligent.
is not superintelligent.
113
337494
2969
05:52
He's pretty smart,
but eventually Dave outwits him
but eventually Dave outwits him
114
340487
3587
它挺聰明的，
但還是比不過人類戴夫，
但還是比不過人類戴夫，
05:56
and manages to switch him off.
115
344098
1849
戴夫可以把 HAL 關掉。
06:01
But we might not be so lucky.
116
349828
1619
但我們可能就沒有這麽幸運了。
06:08
So what are we going to do?
117
356193
1592
那我們應該怎麽辦呢？
06:12
I'm trying to redefine AI
118
360371
2601
我想要重新定義人工智慧，
06:14
to get away from this classical notion
119
362996
2061
不再囿於傳統的概念：
06:17
of machines that intelligently
pursue objectives.
pursue objectives.
120
365081
4567
能明智地達成目標的機器。
06:22
There are three principles involved.
121
370712
1798
新的定義涉及三條原則。
06:24
The first one is a principle
of altruism, if you like,
of altruism, if you like,
122
372534
3289
第一個原則是利他主義原則，
06:27
that the robot's only objective
123
375847
3262
也就是說，機器的唯一目標
06:31
is to maximize the realization
of human objectives,
of human objectives,
124
379133
4246
就是要最大化地實現
人類的目標、人類的價值。
人類的目標、人類的價值。
06:35
of human values.
125
383403
1390
06:36
And by values here I don't mean
touchy-feely, goody-goody values.
touchy-feely, goody-goody values.
126
384817
3330
這種價值不是指多愁善感
或者假裝乖巧，
或者假裝乖巧，
06:40
I just mean whatever it is
that the human would prefer
that the human would prefer
127
388171
3787
而是指人類所嚮往、追求的生活，
無論現狀如何。
無論現狀如何。
06:43
their life to be like.
128
391982
1343
06:47
And so this actually violates Asimov's law
129
395364
2309
事實上，這樣就違反了艾西莫夫定律，
06:49
that the robot has to protect
its own existence.
its own existence.
130
397697
2329
定律裡的機器人必須維護自己的生存。
06:52
It has no interest in preserving
its existence whatsoever.
its existence whatsoever.
131
400050
3723
而在這條原則裡
機器對自身生存與否毫不關心。
機器對自身生存與否毫不關心。
06:57
The second law is a law
of humility, if you like.
of humility, if you like.
132
405420
3768
第二個原則，不妨稱之為謙遜原則。
07:01
And this turns out to be really
important to make robots safe.
important to make robots safe.
133
409974
3743
這一條對製造出安全的機器人十分重要。
07:05
It says that the robot does not know
134
413741
3142
它是指機器人不知道人類的價值是什麽，
07:08
what those human values are,
135
416907
2028
07:10
so it has to maximize them,
but it doesn't know what they are.
but it doesn't know what they are.
136
418959
3178
它只知道將該價值最大化，
但卻不知道該價值究竟是什麽。
但卻不知道該價值究竟是什麽。
07:15
And that avoids this problem
of single-minded pursuit
of single-minded pursuit
137
423254
2626
這就避免了「追求單一目的
而不知變通」的現象。
而不知變通」的現象。
07:17
of an objective.
138
425904
1212
07:19
This uncertainty turns out to be crucial.
139
427140
2172
這種不確定性就變得很重要了。
為了對我們有益，
07:21
Now, in order to be useful to us,
140
429726
1639
07:23
it has to have some idea of what we want.
141
431389
2731
機械就得大概明白我們想要什麽。
07:27
It obtains that information primarily
by observation of human choices,
by observation of human choices,
142
435223
5427
它要獲取這類信息，主要是
透過觀察人類的決策，
透過觀察人類的決策，
07:32
so our own choices reveal information
143
440674
2801
所以我們的決策會揭露
我們生活的意願，
我們生活的意願，
07:35
about what it is that we prefer
our lives to be like.
our lives to be like.
144
443499
3300
07:40
So those are the three principles.
145
448632
1683
所以，這三條原則，
07:42
Let's see how that applies
to this question of:
to this question of:
146
450339
2318
讓我們來看看要如何
應用到圖靈所說的問題：
應用到圖靈所說的問題：
07:44
"Can you switch the machine off?"
as Turing suggested.
as Turing suggested.
147
452681
2789
「你能不能將機器關掉？」
07:49
So here's a PR2 robot.
148
457073
2120
這是 PR2 機器人，
07:51
This is one that we have in our lab,
149
459217
1821
這是我們實驗室裡的其中一台，
07:53
and it has a big red "off" switch
right on the back.
right on the back.
150
461062
2903
它的背面有一個大大的紅色開關。
07:56
The question is: Is it
going to let you switch it off?
going to let you switch it off?
151
464541
2615
那問題來了：它會讓你把它關掉嗎？
07:59
If we do it the classical way,
152
467180
1465
如果我們用傳統的定義製造它，
08:00
we give it the objective of, "Fetch
the coffee, I must fetch the coffee,
the coffee, I must fetch the coffee,
153
468669
3482
我們給它一個「去拿咖啡」的目標，
它會想：「我必須去拿咖啡，
它會想：「我必須去拿咖啡，
08:04
I can't fetch the coffee if I'm dead,"
154
472175
2580
但如果我死了，就不能拿咖啡了。」
08:06
so obviously the PR2
has been listening to my talk,
has been listening to my talk,
155
474779
3341
看來， PR2 聽過我的演講了，
08:10
and so it says, therefore,
"I must disable my 'off' switch,
"I must disable my 'off' switch,
156
478144
3753
因此它說：「我必須讓自己的開關失靈，
08:14
and probably taser all the other
people in Starbucks
people in Starbucks
157
482976
2694
可能還要通過電擊把那些在
星巴克裡干擾我的人都擊暈。」
星巴克裡干擾我的人都擊暈。」
08:17
who might interfere with me."
158
485694
1560
08:19
(Laughter)
159
487278
2062
（笑聲）
08:21
So this seems to be inevitable, right?
160
489364
2153
這無法避免，對吧？
08:23
This kind of failure mode
seems to be inevitable,
seems to be inevitable,
161
491541
2398
這種失敗看起來是必然的，
08:25
and it follows from having
a concrete, definite objective.
a concrete, definite objective.
162
493963
3543
因為機器人會遵循一個
十分明確的目標。
十分明確的目標。
08:30
So what happens if the machine
is uncertain about the objective?
is uncertain about the objective?
163
498812
3144
那如果機器對目標
不那麽確定會發生什麽呢？
不那麽確定會發生什麽呢？
08:33
Well, it reasons in a different way.
164
501980
2127
那它的思路就不一樣了。
08:36
It says, "OK, the human
might switch me off,
might switch me off,
165
504131
2424
它會說：「好的，人類可能會把我關掉，
08:39
but only if I'm doing something wrong.
166
507144
1866
但只有我做錯事了，才會把我關掉。
沒錯，我真的不知道什麽才是錯，
08:41
Well, I don't really know what wrong is,
167
509747
2475
08:44
but I know that I don't want to do it."
168
512246
2044
但我知道我不該做錯的事。」
這就是第一和第二原則。
08:46
So that's the first and second
principles right there.
principles right there.
169
514314
3010
「所以我應該讓人類把我關掉。」
08:49
"So I should let the human switch me off."
170
517348
3359
08:53
And in fact you can calculate
the incentive that the robot has
the incentive that the robot has
171
521721
3956
事實上你可以推斷出機器人為了
允許讓人類關掉它所包含的動機，
允許讓人類關掉它所包含的動機，
08:57
to allow the human to switch it off,
172
525701
2493
09:00
and it's directly tied to the degree
173
528218
1914
而且這與根本目標的
不確定性程度直接相關。
不確定性程度直接相關。
09:02
of uncertainty about
the underlying objective.
the underlying objective.
174
530156
2746
09:05
And then when the machine is switched off,
175
533977
2949
當機器被關閉後，
09:08
that third principle comes into play.
176
536950
1805
第三條原則就起作用了。
09:10
It learns something about the objectives
it should be pursuing,
it should be pursuing,
177
538779
3062
機器開始學習它應追求的目標，
09:13
because it learns that
what it did wasn't right.
what it did wasn't right.
178
541865
2533
因為它知道它剛才做的事是不對的。
09:16
In fact, we can, with suitable use
of Greek symbols,
of Greek symbols,
179
544422
3570
實際上，我們可以適當地
使用些希臘字母，
使用些希臘字母，
09:20
as mathematicians usually do,
180
548016
2131
就像數學家們經常做的那樣，
09:22
we can actually prove a theorem
181
550171
1984
直接證明這一個理論：這樣的
機器人對人類是絕對有利的。
機器人對人類是絕對有利的。
09:24
that says that such a robot
is provably beneficial to the human.
is provably beneficial to the human.
182
552179
3553
09:27
You are provably better off
with a machine that's designed in this way
with a machine that's designed in this way
183
555756
3803
可以證明如此設計出來的機器人，
對我們的生活是是有益的。
對我們的生活是是有益的。
09:31
than without it.
184
559583
1246
09:33
So this is a very simple example,
but this is the first step
but this is the first step
185
561237
2906
這個例子很簡單，
但它是我們嘗試實現
能與人類和諧共處的 AI 的第一步。
能與人類和諧共處的 AI 的第一步。
09:36
in what we're trying to do
with human-compatible AI.
with human-compatible AI.
186
564167
3903
09:42
Now, this third principle,
187
570657
3257
現在來看第三個原則，
09:45
I think is the one that you're probably
scratching your head over.
scratching your head over.
188
573938
3112
我知道各位可能還在為
這一個原則傷腦筋。
這一個原則傷腦筋。
09:49
You're probably thinking, "Well,
you know, I behave badly.
you know, I behave badly.
189
577074
3239
你可能會想：「你懂的，
我行為舉止比較差勁。
我行為舉止比較差勁。
09:52
I don't want my robot to behave like me.
190
580337
2929
我的機器人可不能被我帶壞。
09:55
I sneak down in the middle of the night
and take stuff from the fridge.
and take stuff from the fridge.
191
583290
3434
我有時後會大半夜偷偷摸摸地
從冰箱裡找東西吃，
從冰箱裡找東西吃，
09:58
I do this and that."
192
586748
1168
東瞅瞅，西摸摸。」
09:59
There's all kinds of things
you don't want the robot doing.
you don't want the robot doing.
193
587940
2797
有各種各樣的事
你是不希望機器人去做的。
你是不希望機器人去做的。
但實際上不是那樣。
10:02
But in fact, it doesn't
quite work that way.
quite work that way.
194
590761
2071
10:04
Just because you behave badly
195
592856
2155
你行為不檢，
不代表機器人就得有樣學樣。
10:07
doesn't mean the robot
is going to copy your behavior.
is going to copy your behavior.
196
595035
2623
它會去嘗試理解你做事的動機，
10:09
It's going to understand your motivations
and maybe help you resist them,
and maybe help you resist them,
197
597682
3910
而且可能會在合適的情況下
幫助你、制止你。
幫助你、制止你。
10:13
if appropriate.
198
601616
1320
但這仍然十分困難。
10:16
But it's still difficult.
199
604206
1464
10:18
What we're trying to do, in fact,
200
606302
2545
實際上，我們是要讓機器
10:20
is to allow machines to predict
for any person and for any possible life
for any person and for any possible life
201
608871
5796
為任何人、任何一種
可能的生活去預測：
可能的生活去預測：
他們更想怎樣？更想要什麽？
10:26
that they could live,
202
614691
1161
10:27
and the lives of everybody else:
203
615876
1597
10:29
Which would they prefer?
204
617497
2517
這涉及到諸多困難，
10:34
And there are many, many
difficulties involved in doing this;
difficulties involved in doing this;
205
622061
2954
10:37
I don't expect that this
is going to get solved very quickly.
is going to get solved very quickly.
206
625039
2932
我不認為這會很快地就被解決。
10:39
The real difficulties, in fact, are us.
207
627995
2643
實際上，真正的困難是我們自己。
10:44
As I have already mentioned,
we behave badly.
we behave badly.
208
632149
3117
就像我剛說的那樣，
我們做事不守規矩。
我們做事不守規矩。
10:47
In fact, some of us are downright nasty.
209
635290
2321
我們當中就有人是非常惡劣的。
10:50
Now the robot, as I said,
doesn't have to copy the behavior.
doesn't have to copy the behavior.
210
638431
3052
如前所說，機器人
未必得要複製那些行為。
未必得要複製那些行為。
10:53
The robot does not have
any objective of its own.
any objective of its own.
211
641507
2791
機器人沒有自己的目標，
10:56
It's purely altruistic.
212
644322
1737
它是完全利他的。
10:59
And it's not designed just to satisfy
the desires of one person, the user,
the desires of one person, the user,
213
647293
5221
它的誕生不僅僅是為了去滿足
某一個人、某一個用戶的欲望，
某一個人、某一個用戶的欲望，
11:04
but in fact it has to respect
the preferences of everybody.
the preferences of everybody.
214
652538
3138
而是去尊重所有人的意願。
11:09
So it can deal with a certain
amount of nastiness,
amount of nastiness,
215
657263
2570
所以它懂得抵制一些惡劣的行為，
11:11
and it can even understand
that your nastiness, for example,
that your nastiness, for example,
216
659857
3701
它甚至能理解你為什麼惡劣，比如說，
11:15
you may take bribes as a passport official
217
663582
2671
如果你是一個邊境護照官員，
你可能會收取賄賂，
你可能會收取賄賂，
11:18
because you need to feed your family
and send your kids to school.
and send your kids to school.
218
666277
3812
因為你得養家、供孩子們上學。
機器人能理解這一點，
但不代表它也會學你偷錢，
但不代表它也會學你偷錢，
11:22
It can understand that;
it doesn't mean it's going to steal.
it doesn't mean it's going to steal.
219
670113
2906
11:25
In fact, it'll just help you
send your kids to school.
send your kids to school.
220
673043
2679
它反而會幫助你去供孩子們上學。
11:28
We are also computationally limited.
221
676976
3012
我們的計算能力也是有限的。
11:32
Lee Sedol is a brilliant Go player,
222
680012
2505
李世石是一個傑出的圍棋大師，
11:34
but he still lost.
223
682541
1325
但他還是輸了。
11:35
So if we look at his actions,
he took an action that lost the game.
he took an action that lost the game.
224
683890
4239
如果我們仔細觀察他的棋路，
他下錯了那幾步以致輸棋，
他下錯了那幾步以致輸棋，
11:40
That doesn't mean he wanted to lose.
225
688153
2161
但這不意味著他想要輸。
11:43
So to understand his behavior,
226
691340
2040
所以要理解他的行為，
11:45
we actually have to invert
through a model of human cognition
through a model of human cognition
227
693404
3644
我們得從人類認知的模型回推過來，
11:49
that includes our computational
limitations -- a very complicated model.
limitations -- a very complicated model.
228
697072
4977
它包含了我們計算能力上的局限，
是一個很覆雜的模型。
11:54
But it's still something
that we can work on understanding.
that we can work on understanding.
229
702073
2993
但我們仍然可以嘗試去理解。
11:57
Probably the most difficult part,
from my point of view as an AI researcher,
from my point of view as an AI researcher,
230
705876
4320
可能對於我這樣的 AI 研究人員來說，
12:02
is the fact that there are lots of us,
231
710220
2575
最大的困難是，人有很多種，
12:06
and so the machine has to somehow
trade off, weigh up the preferences
trade off, weigh up the preferences
232
714294
3581
所以機器必須想辦法去協調、
權衡不同人之間的喜好、需求，
權衡不同人之間的喜好、需求，
12:09
of many different people,
233
717899
2225
而要做到這一點有多種不同的方法。
12:12
and there are different ways to do that.
234
720148
1906
12:14
Economists, sociologists,
moral philosophers have understood that,
moral philosophers have understood that,
235
722078
3689
經濟學家、社會學家、
道德哲學家都理解這一點，
道德哲學家都理解這一點，
12:17
and we are actively
looking for collaboration.
looking for collaboration.
236
725791
2455
我們正積極地尋求合作。
12:20
Let's have a look and see what happens
when you get that wrong.
when you get that wrong.
237
728270
3251
讓我們來看看，如果我們把這一步
走錯了會怎麽樣。
走錯了會怎麽樣。
12:23
So you can have
a conversation, for example,
a conversation, for example,
238
731545
2133
比如說，你可能會與你的
人工智慧助理有這樣的對話，
人工智慧助理有這樣的對話，
12:25
with your intelligent personal assistant
239
733702
1944
12:27
that might be available
in a few years' time.
in a few years' time.
240
735670
2285
這樣的人工智慧可能幾年內就會出現。
12:29
Think of a Siri on steroids.
241
737979
2524
可以把它想成是強化版的 Siri 。
12:33
So Siri says, "Your wife called
to remind you about dinner tonight."
to remind you about dinner tonight."
242
741627
4322
Siri 對你說：「你老婆打電話
提醒你別忘了今天的晚宴。」
提醒你別忘了今天的晚宴。」
12:38
And of course, you've forgotten.
"What? What dinner?
"What? What dinner?
243
746616
2508
當然你早就忘了這回事：
「什麽？什麽晚宴？你在說什麽？」
12:41
What are you talking about?"
244
749148
1425
12:42
"Uh, your 20th anniversary at 7pm."
245
750597
3746
「呃.....今晚 7 點
慶祝結婚 20 周年。」
慶祝結婚 20 周年。」
12:48
"I can't do that. I'm meeting
with the secretary-general at 7:30.
with the secretary-general at 7:30.
246
756915
3719
「我可去不了，
我晚上 7 點半要見秘書長。
我晚上 7 點半要見秘書長。
12:52
How could this have happened?"
247
760658
1692
怎麽會這樣呢？」
12:54
"Well, I did warn you, but you overrode
my recommendation."
my recommendation."
248
762374
4660
「呃，我可是提醒過你的，
但你沒有理會我的建議。」
但你沒有理會我的建議。」
13:00
"Well, what am I going to do?
I can't just tell him I'm too busy."
I can't just tell him I'm too busy."
249
768146
3328
「我該怎麽辦呢？我可不能跟秘書長說
我有事，沒空見他。」
我有事，沒空見他。」
13:04
"Don't worry. I arranged
for his plane to be delayed."
for his plane to be delayed."
250
772490
3281
「別擔心。我已經安排了，
讓他的航班延誤。」
讓他的航班延誤。」
13:07
(Laughter)
251
775795
1682
（笑聲）
13:10
"Some kind of computer malfunction."
252
778249
2101
「用某種電腦故障。」
13:12
(Laughter)
253
780374
1212
（笑聲）
13:13
"Really? You can do that?"
254
781610
1617
「真的嗎？這個你也能做到？」
13:16
"He sends his profound apologies
255
784400
2179
「秘書長很不好意思，跟你道歉，
13:18
and looks forward to meeting you
for lunch tomorrow."
for lunch tomorrow."
256
786603
2555
並邀請你明天中午吃飯。」
13:21
(Laughter)
257
789182
1299
（笑聲）
13:22
So the values here --
there's a slight mistake going on.
there's a slight mistake going on.
258
790505
4403
所以這裡談的價值觀就有點問題了，
13:26
This is clearly following my wife's values
259
794932
3009
這顯然是在遵循我老婆的價值觀，
13:29
which is "Happy wife, happy life."
260
797965
2069
也就是「老婆開心，生活舒心」。
13:32
(Laughter)
261
800058
1583
（笑聲）
13:33
It could go the other way.
262
801665
1444
它也有可能發展成另一種情況。
13:35
You could come home
after a hard day's work,
after a hard day's work,
263
803821
2201
你忙碌一天，回到家裏，
13:38
and the computer says, "Long day?"
264
806046
2195
電腦對你說：「今天很忙喔？」
13:40
"Yes, I didn't even have time for lunch."
265
808265
2288
「是啊，我連午飯都沒來得及吃。」
13:42
"You must be very hungry."
266
810577
1282
「那你一定很餓了吧。」
13:43
"Starving, yeah.
Could you make some dinner?"
Could you make some dinner?"
267
811883
2646
「快餓暈了。你能做點晚飯嗎？」
13:48
"There's something I need to tell you."
268
816070
2090
「有一件事我得告訴你。」
13:50
(Laughter)
269
818184
1155
（笑聲）
13:52
"There are humans in South Sudan
who are in more urgent need than you."
who are in more urgent need than you."
270
820193
4905
「南蘇丹人民的情況
比你更緊急，更需要照顧。」
比你更緊急，更需要照顧。」
13:57
(Laughter)
271
825122
1104
（笑聲）
13:58
"So I'm leaving. Make your own dinner."
272
826250
2075
「所以我要走了。你自己做飯去吧。」
14:00
(Laughter)
273
828349
2000
（笑聲）
我們得解決這類的問題，
14:02
So we have to solve these problems,
274
830823
1739
14:04
and I'm looking forward
to working on them.
to working on them.
275
832586
2515
我也很期待能解決這樣的問題。
我們有理由感到樂觀。
14:07
There are reasons for optimism.
276
835125
1843
14:08
One reason is,
277
836992
1159
理由之一是，
14:10
there is a massive amount of data.
278
838175
1868
我們有大量的數據資料。
14:12
Because remember -- I said
they're going to read everything
they're going to read everything
279
840067
2794
記住，我說過機器將能夠
閱讀所有人類寫下來的東西。
閱讀所有人類寫下來的東西。
14:14
the human race has ever written.
280
842885
1546
而我們寫下的文字大都類似於
14:16
Most of what we write about
is human beings doing things
is human beings doing things
281
844455
2724
「人類做了一些事情
導致其他人對此感到沮喪」。
導致其他人對此感到沮喪」。
14:19
and other people getting upset about it.
282
847203
1914
14:21
So there's a massive amount
of data to learn from.
of data to learn from.
283
849141
2398
所以機器可以從
大量的數據中去學習。
大量的數據中去學習。
14:23
There's also a very
strong economic incentive
strong economic incentive
284
851563
2236
同時從經濟的角度，
我們也有足夠的動機去做好這件事。
我們也有足夠的動機去做好這件事。
14:27
to get this right.
285
855331
1186
14:28
So imagine your domestic robot's at home.
286
856541
2001
想像一下，你家裡有個居家機器人。
14:30
You're late from work again
and the robot has to feed the kids,
and the robot has to feed the kids,
287
858566
3067
而你又得加班，
機器人得給孩子們做飯，
機器人得給孩子們做飯，
14:33
and the kids are hungry
and there's nothing in the fridge.
and there's nothing in the fridge.
288
861657
2823
孩子們很餓，
但冰箱裡什麽都沒有。
但冰箱裡什麽都沒有。
14:36
And the robot sees the cat.
289
864504
2605
然後機器人看到了家裡的貓。
14:39
(Laughter)
290
867133
1692
（笑聲）
14:40
And the robot hasn't quite learned
the human value function properly,
the human value function properly,
291
868849
4190
機器人還沒學透人類的價值觀。
14:45
so it doesn't understand
292
873063
1251
所以它不知道，
貓的情感價值大於其營養價值。
貓的情感價值大於其營養價值。
14:46
the sentimental value of the cat outweighs
the nutritional value of the cat.
the nutritional value of the cat.
293
874338
4844
14:51
(Laughter)
294
879206
1095
（笑聲）
14:52
So then what happens?
295
880325
1748
接下來會發生什麽事？
14:54
Well, it happens like this:
296
882097
3297
頭版頭條可能會是這樣：
14:57
"Deranged robot cooks kitty
for family dinner."
for family dinner."
297
885418
2964
「瘋狂機器人煮了貓咪當晚餐！」
15:00
That one incident would be the end
of the domestic robot industry.
of the domestic robot industry.
298
888406
4523
這場意外就足以結束
整個居家機器人的產業。
整個居家機器人的產業。
15:04
So there's a huge incentive
to get this right
to get this right
299
892953
3372
所以在我們實現超級 AI 之前，
我們有足夠的動機把它做對做好。
我們有足夠的動機把它做對做好。
15:08
long before we reach
superintelligent machines.
superintelligent machines.
300
896349
2715
15:12
So to summarize:
301
900128
1535
總結來說：
15:13
I'm actually trying to change
the definition of AI
the definition of AI
302
901687
2881
我事實上想要改變人工智慧的定義，
15:16
so that we have provably
beneficial machines.
beneficial machines.
303
904592
2993
這樣我們就可以製造出
對我們有益無害的機器人。
對我們有益無害的機器人。
15:19
And the principles are:
304
907609
1222
這三個原則是：
15:20
machines that are altruistic,
305
908855
1398
機器是利他的，
15:22
that want to achieve only our objectives,
306
910277
2804
只想著實現我們的目標，
15:25
but that are uncertain
about what those objectives are,
about what those objectives are,
307
913105
3116
但它不確定我們的目標是什麽，
15:28
and will watch all of us
308
916245
1998
並且它會觀察我們，
15:30
to learn more about what it is
that we really want.
that we really want.
309
918267
3203
從中學習我們想要的究竟是什麽。
15:34
And hopefully in the process,
we will learn to be better people.
we will learn to be better people.
310
922373
3559
希望在這個過程中，
我們也能學會成為更好的人。
我們也能學會成為更好的人。
15:37
Thank you very much.
311
925956
1191
謝謝大家。
15:39
(Applause)
312
927171
3709
（掌聲）
克里斯安德森：非常有意思，斯圖爾特。
15:42
Chris Anderson: So interesting, Stuart.
313
930904
1868
15:44
We're going to stand here a bit
because I think they're setting up
because I think they're setting up
314
932796
3170
趁工作人員為下一位講者佈置的時候，
我們先站在這裡聊幾句。
15:47
for our next speaker.
315
935990
1151
15:49
A couple of questions.
316
937165
1538
我有幾個問題。
15:50
So the idea of programming in ignorance
seems intuitively really powerful.
seems intuitively really powerful.
317
938727
5453
將「無知」編寫到程式中，
這種思想真的很有衝擊力。
這種思想真的很有衝擊力。
15:56
As you get to superintelligence,
318
944204
1594
當機器人有超級智慧時，
15:57
what's going to stop a robot
319
945822
2258
還有什麽東西能阻檔機器人閱讀書籍，
16:00
reading literature and discovering
this idea that knowledge
this idea that knowledge
320
948104
2852
並了解到：博學比無知要好得多，
16:02
is actually better than ignorance
321
950980
1572
16:04
and still just shifting its own goals
and rewriting that programming?
and rewriting that programming?
322
952576
4218
進而改變它的目標，
重新編寫自己的程式呢？
重新編寫自己的程式呢？
16:09
Stuart Russell: Yes, so we want
it to learn more, as I said,
it to learn more, as I said,
323
957692
6356
斯圖爾特拉塞爾：是的，
我們想要它去學習，就像我說的，
我們想要它去學習，就像我說的，
讓機器人學習我們的目標，
16:16
about our objectives.
324
964072
1287
16:17
It'll only become more certain
as it becomes more correct,
as it becomes more correct,
325
965383
5521
只有在理解得越正確的時候，
它們才會更明確我們要的東西，
它們才會更明確我們要的東西，
16:22
so the evidence is there
326
970928
1945
佐證擺在那裡，
16:24
and it's going to be designed
to interpret it correctly.
to interpret it correctly.
327
972897
2724
並且我們使它能夠正確解讀這些目標。
16:27
It will understand, for example,
that books are very biased
that books are very biased
328
975645
3956
比如說，它能夠從書中的佐證
判斷出那些富含偏見的書，
判斷出那些富含偏見的書，
16:31
in the evidence they contain.
329
979625
1483
16:33
They only talk about kings and princes
330
981132
2397
像是只講述國王、王子，
和男性精英白人之類的書。
和男性精英白人之類的書。
16:35
and elite white male people doing stuff.
331
983553
2800
16:38
So it's a complicated problem,
332
986377
2096
所以這是一個複雜的問題，
16:40
but as it learns more about our objectives
333
988497
3872
但當它更深入地學習我們的目標時，
16:44
it will become more and more useful to us.
334
992393
2063
它會變得越來越有用。
16:46
CA: And you couldn't
just boil it down to one law,
just boil it down to one law,
335
994480
2526
CA：所以它十分複雜，
遠不足以濃縮成一條法則嗎？
遠不足以濃縮成一條法則嗎？
16:49
you know, hardwired in:
336
997030
1650
像是，把這樣的命令燒録進去：
16:50
"if any human ever tries to switch me off,
337
998704
3293
「如果人類想把我關掉，
16:54
I comply. I comply."
338
1002021
1935
我要服從。我要服從。」
16:55
SR: Absolutely not.
339
1003980
1182
SR：絕對不行。
16:57
That would be a terrible idea.
340
1005186
1499
那將是一個很糟糕的主意。
16:58
So imagine that you have
a self-driving car
a self-driving car
341
1006709
2689
試想一下，你有一輛無人駕駛汽車，
17:01
and you want to send your five-year-old
342
1009422
2433
你想讓它送你五歲的孩子去幼稚園。
17:03
off to preschool.
343
1011879
1174
17:05
Do you want your five-year-old
to be able to switch off the car
to be able to switch off the car
344
1013077
3101
你會希望你五歲的孩子
在汽車運行的過程中將它關閉嗎？
在汽車運行的過程中將它關閉嗎？
17:08
while it's driving along?
345
1016202
1213
應該不會吧。
17:09
Probably not.
346
1017439
1159
所以它得理解
17:10
So it needs to understand how rational
and sensible the person is.
and sensible the person is.
347
1018622
4703
下指令的人有多理智、有多講道理。
17:15
The more rational the person,
348
1023349
1676
這個人越理智，
17:17
the more willing you are
to be switched off.
to be switched off.
349
1025049
2103
它就越願意被你關掉。
如果這個人是完全思緒混亂
或者甚至是有惡意的，
或者甚至是有惡意的，
17:19
If the person is completely
random or even malicious,
random or even malicious,
350
1027176
2543
那它就不太願意被你關掉了。
17:21
then you're less willing
to be switched off.
to be switched off.
351
1029743
2512
17:24
CA: All right. Stuart, can I just say,
352
1032279
1866
CA：好吧。斯圖爾特，我得說，
17:26
I really, really hope you
figure this out for us.
figure this out for us.
353
1034169
2314
我真的希望你為我們所有人，
找到解決的辦法。
找到解決的辦法。
很感謝你的演講。
十分精彩。
十分精彩。
17:28
Thank you so much for that talk.
That was amazing.
That was amazing.
354
1036507
2375
SR：謝謝。
CA：謝謝。
17:30
SR: Thank you.
355
1038906
1167
（掌聲）
17:32
(Applause)
356
1040097
1837
ABOUT THE SPEAKER
Stuart Russell - AI expertStuart Russell wrote the standard text on AI; now he thinks deeply on AI's future -- and the future of us humans, too.
Why you should listen
Stuart Russell is a professor (and formerly chair) of Electrical Engineering and Computer Sciences at University of California at Berkeley. His book Artificial Intelligence: A Modern Approach (with Peter Norvig) is the standard text in AI; it has been translated into 13 languages and is used in more than 1,300 universities in 118 countries. His research covers a wide range of topics in artificial intelligence including machine learning, probabilistic reasoning, knowledge representation, planning, real-time decision making, multitarget tracking, computer vision, computational physiology, global seismic monitoring and philosophical foundations.
He also works for the United Nations, developing a new global seismic monitoring system for the nuclear-test-ban treaty. His current concerns include the threat of autonomous weapons and the long-term future of artificial intelligence and its relation to humanity.
Stuart Russell | Speaker | TED.com