TEDMED 2012
Miguel Nicolelis: A monkey that controls a robot with its thoughts. No, really.
Filmed:
Readability: 4.1
1,315,130 views
Can we use our brains to directly control machines? Miguel Nicolelis suggests yes, showing how a clever monkey in the US learned to control a robot arm in Japan purely with its thoughts. The research has big implications for quadraplegic people -- and in fact, it powered the exoskeleton that kicked off the 2014 World Cup.
Miguel Nicolelis - Neuroscientist
Miguel Nicolelis explores the limits of the brain-machine interface. Full bio
Miguel Nicolelis explores the limits of the brain-machine interface. Full bio
Double-click the English transcript below to play the video.
00:16
The kind of neuroscience that I do and my colleagues do
0
330
2851
00:19
is almost like the weatherman.
1
3181
2166
00:21
We are always chasing storms.
2
5347
3516
00:24
We want to see and measure storms -- brainstorms, that is.
3
8863
4883
00:29
And we all talk about brainstorms in our daily lives,
4
13746
2768
00:32
but we rarely see or listen to one.
5
16514
3450
00:35
So I always like to start these talks
6
19964
1634
00:37
by actually introducing you to one of them.
7
21598
2982
00:40
Actually, the first time we recorded more than one neuron --
8
24580
3427
00:43
a hundred brain cells simultaneously --
9
28007
2223
00:46
we could measure the electrical sparks
10
30230
2469
00:48
of a hundred cells in the same animal,
11
32699
2680
00:51
this is the first image we got,
12
35379
1802
00:53
the first 10 seconds of this recording.
13
37181
2315
00:55
So we got a little snippet of a thought,
14
39496
3351
00:58
and we could see it in front of us.
15
42847
2905
01:01
I always tell the students
16
45752
1012
01:02
that we could also call neuroscientists some sort of astronomer,
17
46764
4106
01:06
because we are dealing with a system
18
50870
1626
01:08
that is only comparable in terms of number of cells
19
52496
2917
01:11
to the number of galaxies that we have in the universe.
20
55413
2936
01:14
And here we are, out of billions of neurons,
21
58349
3030
01:17
just recording, 10 years ago, a hundred.
22
61379
2818
01:20
We are doing a thousand now.
23
64197
1583
01:21
And we hope to understand something fundamental about our human nature.
24
65780
5400
01:27
Because, if you don't know yet,
25
71180
1932
01:29
everything that we use to define what human nature is comes from these storms,
26
73112
5250
01:34
comes from these storms that roll over the hills and valleys of our brains
27
78362
4651
01:38
and define our memories, our beliefs,
28
83013
3885
01:42
our feelings, our plans for the future.
29
86898
2700
01:45
Everything that we ever do,
30
89598
2398
01:47
everything that every human has ever done, do or will do,
31
91996
5067
01:52
requires the toil of populations of neurons producing these kinds of storms.
32
97063
5434
01:58
And the sound of a brainstorm, if you've never heard one,
33
102497
2483
02:00
is somewhat like this.
34
104980
3349
02:04
You can put it louder if you can.
35
108329
3146
02:07
My son calls this "making popcorn while listening to a badly-tuned A.M. station."
36
111475
6403
02:13
This is a brain.
37
117878
1485
02:15
This is what happens when you route these electrical storms to a loudspeaker
38
119363
3434
02:18
and you listen to a hundred brain cells firing,
39
122797
2866
02:21
your brain will sound like this -- my brain, any brain.
40
125663
4622
02:26
And what we want to do as neuroscientists in this time
41
130285
3762
02:29
is to actually listen to these symphonies, these brain symphonies,
42
134047
5350
02:35
and try to extract from them the messages they carry.
43
139397
3400
02:38
In particular, about 12 years ago
44
142797
2851
02:41
we created a preparation that we named brain-machine interfaces.
45
145648
3048
02:44
And you have a scheme here that describes how it works.
46
148696
2702
02:47
The idea is, let's have some sensors that listen to these storms, this electrical firing,
47
151398
5566
02:52
and see if you can, in the same time that it takes
48
156964
3082
02:55
for this storm to leave the brain and reach the legs or the arms of an animal --
49
160046
4969
03:00
about half a second --
50
165015
2864
03:03
let's see if we can read these signals,
51
167879
2351
03:06
extract the motor messages that are embedded in it,
52
170230
3400
03:09
translate it into digital commands
53
173630
2272
03:11
and send it to an artificial device
54
175902
1886
03:13
that will reproduce the voluntary motor wheel of that brain in real time.
55
177788
5893
03:19
And see if we can measure how well we can translate that message
56
183681
3848
03:23
when we compare to the way the body does that.
57
187529
3518
03:26
And if we can actually provide feedback,
58
191047
2866
03:29
sensory signals that go back from this robotic, mechanical, computational actuator
59
193913
5734
03:35
that is now under the control of the brain,
60
199647
2251
03:37
back to the brain,
61
201898
1311
03:39
how the brain deals with that,
62
203209
2121
03:41
of receiving messages from an artificial piece of machinery.
63
205330
4901
03:46
And that's exactly what we did 10 years ago.
64
210231
2321
03:48
We started with a superstar monkey called Aurora
65
212552
2961
03:51
that became one of the superstars of this field.
66
215513
2468
03:53
And Aurora liked to play video games.
67
217981
2299
03:56
As you can see here,
68
220280
1373
03:57
she likes to use a joystick, like any one of us, any of our kids, to play this game.
69
221653
4944
04:02
And as a good primate, she even tries to cheat before she gets the right answer.
70
226597
4671
04:07
So even before a target appears that she's supposed to cross
71
231268
4283
04:11
with the cursor that she's controlling with this joystick,
72
235551
2850
04:14
Aurora is trying to find the target, no matter where it is.
73
238401
3951
04:18
And if she's doing that,
74
242352
1469
04:19
because every time she crosses that target with the little cursor,
75
243821
3314
04:23
she gets a drop of Brazilian orange juice.
76
247135
2950
04:25
And I can tell you, any monkey will do anything for you
77
250085
2950
04:28
if you get a little drop of Brazilian orange juice.
78
253035
3100
04:32
Actually any primate will do that.
79
256135
2731
04:34
Think about that.
80
258866
1334
04:36
Well, while Aurora was playing this game, as you saw,
81
260200
3400
04:39
and doing a thousand trials a day
82
263600
2435
04:41
and getting 97 percent correct and 350 milliliters of orange juice,
83
266035
3883
04:45
we are recording the brainstorms that are produced in her head
84
269918
3399
04:49
and sending them to a robotic arm
85
273317
1647
04:50
that was learning to reproduce the movements that Aurora was making.
86
274964
3871
04:54
Because the idea was to actually turn on this brain-machine interface
87
278835
3783
04:58
and have Aurora play the game just by thinking,
88
282618
4700
05:03
without interference of her body.
89
287318
2617
05:05
Her brainstorms would control an arm
90
289935
2916
05:08
that would move the cursor and cross the target.
91
292851
2709
05:11
And to our shock, that's exactly what Aurora did.
92
295560
3191
05:14
She played the game without moving her body.
93
298751
4200
05:18
So every trajectory that you see of the cursor now,
94
302951
2237
05:21
this is the exact first moment she got that.
95
305188
3212
05:24
That's the exact first moment
96
308400
1784
05:26
a brain intention was liberated from the physical domains of a body of a primate
97
310184
6767
05:32
and could act outside, in that outside world,
98
316951
3700
05:36
just by controlling an artificial device.
99
320651
2966
05:39
And Aurora kept playing the game, kept finding the little target
100
323617
4917
05:44
and getting the orange juice that she wanted to get, that she craved for.
101
328534
3917
05:48
Well, she did that because she, at that time, had acquired a new arm.
102
332451
6701
05:55
The robotic arm that you see moving here 30 days later,
103
339152
2963
05:58
after the first video that I showed to you,
104
342115
2686
06:00
is under the control of Aurora's brain
105
344801
2650
06:03
and is moving the cursor to get to the target.
106
347451
3168
06:06
And Aurora now knows that she can play the game with this robotic arm,
107
350619
3899
06:10
but she has not lost the ability to use her biological arms to do what she pleases.
108
354518
5716
06:16
She can scratch her back, she can scratch one of us, she can play another game.
109
360234
4067
06:20
By all purposes and means,
110
364301
1600
06:21
Aurora's brain has incorporated that artificial device
111
365901
4116
06:25
as an extension of her body.
112
370017
2750
06:28
The model of the self that Aurora had in her mind
113
372767
3533
06:32
has been expanded to get one more arm.
114
376300
4084
06:36
Well, we did that 10 years ago.
115
380384
2350
06:38
Just fast forward 10 years.
116
382734
2833
06:41
Just last year we realized that you don't even need to have a robotic device.
117
385567
4983
06:46
You can just build a computational body, an avatar, a monkey avatar.
118
390550
5484
06:51
And you can actually use it for our monkeys to either interact with them,
119
396034
4250
06:56
or you can train them to assume in a virtual world
120
400284
4439
07:00
the first-person perspective of that avatar
121
404723
3044
07:03
and use her brain activity to control the movements of the avatar's arms or legs.
122
407767
5651
07:09
And what we did basically was to train the animals
123
413418
2766
07:12
to learn how to control these avatars
124
416184
3050
07:15
and explore objects that appear in the virtual world.
125
419234
3899
07:19
And these objects are visually identical,
126
423133
2301
07:21
but when the avatar crosses the surface of these objects,
127
425434
3883
07:25
they send an electrical message that is proportional to the microtactile texture of the object
128
429317
6400
07:31
that goes back directly to the monkey's brain,
129
435717
4016
07:35
informing the brain what it is the avatar is touching.
130
439733
5052
07:40
And in just four weeks, the brain learns to process this new sensation
131
444785
4765
07:45
and acquires a new sensory pathway -- like a new sense.
132
449550
6434
07:51
And you truly liberate the brain now
133
455984
2416
07:54
because you are allowing the brain to send motor commands to move this avatar.
134
458400
4384
07:58
And the feedback that comes from the avatar is being processed directly by the brain
135
462784
5000
08:03
without the interference of the skin.
136
467784
2433
08:06
So what you see here is this is the design of the task.
137
470217
2534
08:08
You're going to see an animal basically touching these three targets.
138
472751
4250
08:12
And he has to select one because only one carries the reward,
139
477001
4349
08:17
the orange juice that they want to get.
140
481350
1867
08:19
And he has to select it by touch using a virtual arm, an arm that doesn't exist.
141
483217
5633
08:24
And that's exactly what they do.
142
488850
2000
08:26
This is a complete liberation of the brain
143
490850
3435
08:30
from the physical constraints of the body and the motor in a perceptual task.
144
494285
4282
08:34
The animal is controlling the avatar to touch the targets.
145
498567
4167
08:38
And he's sensing the texture by receiving an electrical message directly in the brain.
146
502734
5651
08:44
And the brain is deciding what is the texture associated with the reward.
147
508385
3883
08:48
The legends that you see in the movie don't appear for the monkey.
148
512268
3832
08:52
And by the way, they don't read English anyway,
149
516100
2484
08:54
so they are here just for you to know that the correct target is shifting position.
150
518584
5216
08:59
And yet, they can find them by tactile discrimination,
151
523800
3934
09:03
and they can press it and select it.
152
527734
3217
09:06
So when we look at the brains of these animals,
153
530951
2682
09:09
on the top panel you see the alignment of 125 cells
154
533633
3667
09:13
showing what happens with the brain activity, the electrical storms,
155
537300
4201
09:17
of this sample of neurons in the brain
156
541501
2067
09:19
when the animal is using a joystick.
157
543568
2116
09:21
And that's a picture that every neurophysiologist knows.
158
545684
2600
09:24
The basic alignment shows that these cells are coding for all possible directions.
159
548284
5183
09:29
The bottom picture is what happens when the body stops moving
160
553467
5683
09:35
and the animal starts controlling either a robotic device or a computational avatar.
161
559150
6134
09:41
As fast as we can reset our computers,
162
565284
3066
09:44
the brain activity shifts to start representing this new tool,
163
568350
5818
09:50
as if this too was a part of that primate's body.
164
574168
5250
09:55
The brain is assimilating that too, as fast as we can measure.
165
579418
4715
10:00
So that suggests to us that our sense of self
166
584133
3618
10:03
does not end at the last layer of the epithelium of our bodies,
167
587751
4150
10:07
but it ends at the last layer of electrons of the tools that we're commanding with our brains.
168
591901
5718
10:13
Our violins, our cars, our bicycles, our soccer balls, our clothing --
169
597619
4764
10:18
they all become assimilated by this voracious, amazing, dynamic system called the brain.
170
602383
6851
10:25
How far can we take it?
171
609234
1699
10:26
Well, in an experiment that we ran a few years ago, we took this to the limit.
172
610933
4218
10:31
We had an animal running on a treadmill
173
615151
2482
10:33
at Duke University on the East Coast of the United States,
174
617633
2267
10:35
producing the brainstorms necessary to move.
175
619900
2700
10:38
And we had a robotic device, a humanoid robot,
176
622600
4091
10:42
in Kyoto, Japan at ATR Laboratories
177
626691
2394
10:44
that was dreaming its entire life to be controlled by a brain,
178
629085
6094
10:51
a human brain, or a primate brain.
179
635179
3273
10:54
What happens here is that the brain activity that generated the movements in the monkey
180
638452
4598
10:58
was transmitted to Japan and made this robot walk
181
643050
3467
11:02
while footage of this walking was sent back to Duke,
182
646517
4067
11:06
so that the monkey could see the legs of this robot walking in front of her.
183
650584
5233
11:11
So she could be rewarded, not by what her body was doing
184
655817
4067
11:15
but for every correct step of the robot on the other side of the planet
185
659884
4961
11:20
controlled by her brain activity.
186
664845
2609
11:23
Funny thing, that round trip around the globe took 20 milliseconds less
187
667454
7118
11:30
than it takes for that brainstorm to leave its head, the head of the monkey,
188
674572
4150
11:34
and reach its own muscle.
189
678722
3870
11:38
The monkey was moving a robot that was six times bigger, across the planet.
190
682592
6030
11:44
This is one of the experiments in which that robot was able to walk autonomously.
191
688622
6400
11:50
This is CB1 fulfilling its dream in Japan
192
695022
5267
11:56
under the control of the brain activity of a primate.
193
700289
3700
11:59
So where are we taking all this?
194
703989
1989
12:01
What are we going to do with all this research,
195
705978
2343
12:04
besides studying the properties of this dynamic universe that we have between our ears?
196
708321
5668
12:09
Well the idea is to take all this knowledge and technology
197
713989
4833
12:14
and try to restore one of the most severe neurological problems that we have in the world.
198
718822
5484
12:20
Millions of people have lost the ability to translate these brainstorms
199
724306
4583
12:24
into action, into movement.
200
728889
2116
12:26
Although their brains continue to produce those storms and code for movements,
201
731005
5234
12:32
they cannot cross a barrier that was created by a lesion on the spinal cord.
202
736239
5167
12:37
So our idea is to create a bypass,
203
741406
2450
12:39
is to use these brain-machine interfaces to read these signals,
204
743856
4032
12:43
larger-scale brainstorms that contain the desire to move again,
205
747888
4050
12:47
bypass the lesion using computational microengineering
206
751938
3969
12:51
and send it to a new body, a whole body called an exoskeleton,
207
755907
7114
12:58
a whole robotic suit that will become the new body of these patients.
208
763021
5567
13:04
And you can see an image produced by this consortium.
209
768588
4126
13:08
This is a nonprofit consortium called the Walk Again Project
210
772714
4059
13:12
that is putting together scientists from Europe,
211
776773
2783
13:15
from here in the United States, and in Brazil
212
779556
1865
13:17
together to work to actually get this new body built --
213
781421
4517
13:21
a body that we believe, through the same plastic mechanisms
214
785938
3334
13:25
that allow Aurora and other monkeys to use these tools through a brain-machine interface
215
789272
5802
13:30
and that allows us to incorporate the tools that we produce and use in our daily life.
216
795074
5630
13:36
This same mechanism, we hope, will allow these patients,
217
800704
3684
13:40
not only to imagine again the movements that they want to make
218
804388
3768
13:44
and translate them into movements of this new body,
219
808156
3207
13:47
but for this body to be assimilated as the new body that the brain controls.
220
811363
6758
13:54
So I was told about 10 years ago
221
818121
3851
13:57
that this would never happen, that this was close to impossible.
222
821972
5066
14:02
And I can only tell you that as a scientist,
223
827038
2451
14:05
I grew up in southern Brazil in the mid-'60s
224
829489
2986
14:08
watching a few crazy guys telling [us] that they would go to the Moon.
225
832475
5048
14:13
And I was five years old,
226
837523
1461
14:14
and I never understood why NASA didn't hire Captain Kirk and Spock to do the job;
227
838984
4240
14:19
after all, they were very proficient --
228
843224
2432
14:21
but just seeing that as a kid
229
845656
3450
14:25
made me believe, as my grandmother used to tell me,
230
849106
2985
14:27
that "impossible is just the possible
231
852091
1845
14:29
that someone has not put in enough effort to make it come true."
232
853936
3904
14:33
So they told me that it's impossible to make someone walk.
233
857840
3799
14:37
I think I'm going to follow my grandmother's advice.
234
861639
3251
14:40
Thank you.
235
864890
1450
14:42
(Applause)
236
866340
8029
ABOUT THE SPEAKER
Miguel Nicolelis - NeuroscientistMiguel Nicolelis explores the limits of the brain-machine interface.
Why you should listen
At the Nicolelis Laboratory at Duke University, Miguel Nicolelis is best known for pioneering studies in neuronal population coding, Brain Machine Interfaces (BMI) and neuroprosthetics in human patients and non-human primates.His lab's work was seen, famously though a bit too briefly, when a brain-controlled exoskeleton from his lab helped Juliano Pinto, a paraplegic man, kick the first ball at the 2014 World Cup.
But his lab is thinking even bigger. They've developed an integrative approach to studying neurological disorders, including Parkinsons disease and epilepsy. The approach, they hope, will allow the integration of molecular, cellular, systems and behavioral data in the same animal, producing a more complete understanding of the nature of the neurophysiological alterations associated with these disorders. He's the author of the books Beyond Boundaries and The Relativistic Brain.
Miguel was honored as one of Foreign Policy's 2015 Global Thinkers.
Miguel Nicolelis | Speaker | TED.com