TEDxPenn
Vijay Kumar: The future of flying robots
Filmed:
Readability: 4.9
1,780,679 views
At his lab at the University of Pennsylvania, Vijay Kumar and his team have created autonomous aerial robots inspired by honeybees. Their latest breakthrough: Precision Farming, in which swarms of robots map, reconstruct and analyze every plant and piece of fruit in an orchard, providing vital information to farmers that can help improve yields and make water management smarter.
Vijay Kumar - Roboticist
As the dean of the University of Pennsylvania's School of Engineering and Applied Science, Vijay Kumar studies the control and coordination of multi-robot formations. Full bio
As the dean of the University of Pennsylvania's School of Engineering and Applied Science, Vijay Kumar studies the control and coordination of multi-robot formations. Full bio
Double-click the English transcript below to play the video.
00:13
In my lab, we build
autonomous aerial robots
autonomous aerial robots
0
1280
3656
00:16
like the one you see flying here.
1
4960
1880
00:20
Unlike the commercially available drones
that you can buy today,
that you can buy today,
2
8720
3696
00:24
this robot doesn't have any GPS on board.
3
12440
2640
00:28
So without GPS,
4
16160
1216
00:29
it's hard for robots like this
to determine their position.
to determine their position.
5
17400
3280
00:34
This robot uses onboard sensors,
cameras and laser scanners,
cameras and laser scanners,
6
22240
4736
00:39
to scan the environment.
7
27000
1696
00:40
It detects features from the environment,
8
28720
3056
00:43
and it determines where it is
relative to those features,
relative to those features,
9
31800
2736
00:46
using a method of triangulation.
10
34560
2136
00:48
And then it can assemble
all these features into a map,
all these features into a map,
11
36720
3456
00:52
like you see behind me.
12
40200
1736
00:53
And this map then allows the robot
to understand where the obstacles are
to understand where the obstacles are
13
41960
3936
00:57
and navigate in a collision-free manner.
14
45920
2720
01:01
What I want to show you next
15
49160
2096
01:03
is a set of experiments
we did inside our laboratory,
we did inside our laboratory,
16
51280
3216
01:06
where this robot was able
to go for longer distances.
to go for longer distances.
17
54520
3480
01:10
So here you'll see, on the top right,
what the robot sees with the camera.
what the robot sees with the camera.
18
58400
5016
01:15
And on the main screen --
19
63440
1216
01:16
and of course this is sped up
by a factor of four --
by a factor of four --
20
64680
2456
01:19
on the main screen you'll see
the map that it's building.
the map that it's building.
21
67160
2667
01:21
So this is a high-resolution map
of the corridor around our laboratory.
of the corridor around our laboratory.
22
69851
4285
01:26
And in a minute
you'll see it enter our lab,
you'll see it enter our lab,
23
74160
2336
01:28
which is recognizable
by the clutter that you see.
by the clutter that you see.
24
76520
2856
01:31
(Laughter)
25
79400
1016
01:32
But the main point I want to convey to you
26
80440
2007
01:34
is that these robots are capable
of building high-resolution maps
of building high-resolution maps
27
82472
3584
01:38
at five centimeters resolution,
28
86080
2496
01:40
allowing somebody who is outside the lab,
or outside the building
or outside the building
29
88600
4176
01:44
to deploy these
without actually going inside,
without actually going inside,
30
92800
3216
01:48
and trying to infer
what happens inside the building.
what happens inside the building.
31
96040
3760
01:52
Now there's one problem
with robots like this.
with robots like this.
32
100400
2240
01:55
The first problem is it's pretty big.
33
103600
2200
01:58
Because it's big, it's heavy.
34
106120
1680
02:00
And these robots consume
about 100 watts per pound.
about 100 watts per pound.
35
108640
3040
02:04
And this makes for
a very short mission life.
a very short mission life.
36
112360
2280
02:08
The second problem
37
116000
1456
02:09
is that these robots have onboard sensors
that end up being very expensive --
that end up being very expensive --
38
117480
3896
02:13
a laser scanner, a camera
and the processors.
and the processors.
39
121400
3440
02:17
That drives up the cost of this robot.
40
125280
3040
02:21
So we asked ourselves a question:
41
129440
2656
02:24
what consumer product
can you buy in an electronics store
can you buy in an electronics store
42
132120
3776
02:27
that is inexpensive, that's lightweight,
that has sensing onboard and computation?
that has sensing onboard and computation?
43
135920
6280
02:36
And we invented the flying phone.
44
144080
2656
02:38
(Laughter)
45
146760
1936
02:40
So this robot uses a Samsung Galaxy
smartphone that you can buy off the shelf,
smartphone that you can buy off the shelf,
46
148720
6176
02:46
and all you need is an app that you
can download from our app store.
can download from our app store.
47
154920
4016
02:50
And you can see this robot
reading the letters, "TED" in this case,
reading the letters, "TED" in this case,
48
158960
4216
02:55
looking at the corners
of the "T" and the "E"
of the "T" and the "E"
49
163200
2936
02:58
and then triangulating off of that,
flying autonomously.
flying autonomously.
50
166160
3480
03:02
That joystick is just there
to make sure if the robot goes crazy,
to make sure if the robot goes crazy,
51
170720
3256
03:06
Giuseppe can kill it.
52
174000
1416
03:07
(Laughter)
53
175440
1640
03:10
In addition to building
these small robots,
these small robots,
54
178920
3816
03:14
we also experiment with aggressive
behaviors, like you see here.
behaviors, like you see here.
55
182760
4800
03:19
So this robot is now traveling
at two to three meters per second,
at two to three meters per second,
56
187920
5296
03:25
pitching and rolling aggressively
as it changes direction.
as it changes direction.
57
193240
3496
03:28
The main point is we can have
smaller robots that can go faster
smaller robots that can go faster
58
196760
4256
03:33
and then travel in these
very unstructured environments.
very unstructured environments.
59
201040
2960
03:37
And in this next video,
60
205120
2056
03:39
just like you see this bird, an eagle,
gracefully coordinating its wings,
gracefully coordinating its wings,
61
207200
5896
03:45
its eyes and feet
to grab prey out of the water,
to grab prey out of the water,
62
213120
4296
03:49
our robot can go fishing, too.
63
217440
1896
03:51
(Laughter)
64
219360
1496
03:52
In this case, this is a Philly cheesesteak
hoagie that it's grabbing out of thin air.
hoagie that it's grabbing out of thin air.
65
220880
4056
03:56
(Laughter)
66
224960
2400
03:59
So you can see this robot
going at about three meters per second,
going at about three meters per second,
67
227680
3296
04:03
which is faster than walking speed,
coordinating its arms, its claws
coordinating its arms, its claws
68
231000
5136
04:08
and its flight with split-second timing
to achieve this maneuver.
to achieve this maneuver.
69
236160
4120
04:14
In another experiment,
70
242120
1216
04:15
I want to show you
how the robot adapts its flight
how the robot adapts its flight
71
243360
3656
04:19
to control its suspended payload,
72
247040
2376
04:21
whose length is actually larger
than the width of the window.
than the width of the window.
73
249440
3800
04:25
So in order to accomplish this,
74
253680
1696
04:27
it actually has to pitch
and adjust the altitude
and adjust the altitude
75
255400
3696
04:31
and swing the payload through.
76
259120
2320
04:38
But of course we want
to make these even smaller,
to make these even smaller,
77
266920
2296
04:41
and we're inspired
in particular by honeybees.
in particular by honeybees.
78
269240
3016
04:44
So if you look at honeybees,
and this is a slowed down video,
and this is a slowed down video,
79
272280
3256
04:47
they're so small,
the inertia is so lightweight --
the inertia is so lightweight --
80
275560
3720
04:51
(Laughter)
81
279960
1176
04:53
that they don't care --
they bounce off my hand, for example.
they bounce off my hand, for example.
82
281160
3536
04:56
This is a little robot
that mimics the honeybee behavior.
that mimics the honeybee behavior.
83
284720
3160
05:00
And smaller is better,
84
288600
1216
05:01
because along with the small size
you get lower inertia.
you get lower inertia.
85
289840
3536
05:05
Along with lower inertia --
86
293400
1536
05:06
(Robot buzzing, laughter)
87
294960
2856
05:09
along with lower inertia,
you're resistant to collisions.
you're resistant to collisions.
88
297840
2816
05:12
And that makes you more robust.
89
300680
1720
05:15
So just like these honeybees,
we build small robots.
we build small robots.
90
303800
2656
05:18
And this particular one
is only 25 grams in weight.
is only 25 grams in weight.
91
306480
3376
05:21
It consumes only six watts of power.
92
309880
2160
05:24
And it can travel
up to six meters per second.
up to six meters per second.
93
312440
2536
05:27
So if I normalize that to its size,
94
315000
2336
05:29
it's like a Boeing 787 traveling
ten times the speed of sound.
ten times the speed of sound.
95
317360
3640
05:36
(Laughter)
96
324000
2096
05:38
And I want to show you an example.
97
326120
1920
05:40
This is probably the first planned mid-air
collision, at one-twentieth normal speed.
collision, at one-twentieth normal speed.
98
328840
5256
05:46
These are going at a relative speed
of two meters per second,
of two meters per second,
99
334120
2858
05:49
and this illustrates the basic principle.
100
337002
2480
05:52
The two-gram carbon fiber cage around it
prevents the propellers from entangling,
prevents the propellers from entangling,
101
340200
4976
05:57
but essentially the collision is absorbed
and the robot responds to the collisions.
and the robot responds to the collisions.
102
345200
5296
06:02
And so small also means safe.
103
350520
2560
06:05
In my lab, as we developed these robots,
104
353400
2016
06:07
we start off with these big robots
105
355440
1620
06:09
and then now we're down
to these small robots.
to these small robots.
106
357084
2812
06:11
And if you plot a histogram
of the number of Band-Aids we've ordered
of the number of Band-Aids we've ordered
107
359920
3456
06:15
in the past, that sort of tailed off now.
108
363400
2576
06:18
Because these robots are really safe.
109
366000
1960
06:20
The small size has some disadvantages,
110
368760
2456
06:23
and nature has found a number of ways
to compensate for these disadvantages.
to compensate for these disadvantages.
111
371240
4080
06:27
The basic idea is they aggregate
to form large groups, or swarms.
to form large groups, or swarms.
112
375960
4000
06:32
So, similarly, in our lab,
we try to create artificial robot swarms.
we try to create artificial robot swarms.
113
380320
3976
06:36
And this is quite challenging
114
384320
1381
06:37
because now you have to think
about networks of robots.
about networks of robots.
115
385725
3320
06:41
And within each robot,
116
389360
1296
06:42
you have to think about the interplay
of sensing, communication, computation --
of sensing, communication, computation --
117
390680
5616
06:48
and this network then becomes
quite difficult to control and manage.
quite difficult to control and manage.
118
396320
4960
06:54
So from nature we take away
three organizing principles
three organizing principles
119
402160
3296
06:57
that essentially allow us
to develop our algorithms.
to develop our algorithms.
120
405480
3160
07:01
The first idea is that robots
need to be aware of their neighbors.
need to be aware of their neighbors.
121
409640
4536
07:06
They need to be able to sense
and communicate with their neighbors.
and communicate with their neighbors.
122
414200
3440
07:10
So this video illustrates the basic idea.
123
418040
2656
07:12
You have four robots --
124
420720
1296
07:14
one of the robots has actually been
hijacked by a human operator, literally.
hijacked by a human operator, literally.
125
422040
4240
07:19
But because the robots
interact with each other,
interact with each other,
126
427217
2239
07:21
they sense their neighbors,
127
429480
1656
07:23
they essentially follow.
128
431160
1296
07:24
And here there's a single person
able to lead this network of followers.
able to lead this network of followers.
129
432480
5360
07:32
So again, it's not because all the robots
know where they're supposed to go.
know where they're supposed to go.
130
440000
5056
07:37
It's because they're just reacting
to the positions of their neighbors.
to the positions of their neighbors.
131
445080
4320
07:43
(Laughter)
132
451720
4120
07:48
So the next experiment illustrates
the second organizing principle.
the second organizing principle.
133
456280
5240
07:54
And this principle has to do
with the principle of anonymity.
with the principle of anonymity.
134
462920
3800
07:59
Here the key idea is that
135
467400
4296
08:03
the robots are agnostic
to the identities of their neighbors.
to the identities of their neighbors.
136
471720
4240
08:08
They're asked to form a circular shape,
137
476440
2616
08:11
and no matter how many robots
you introduce into the formation,
you introduce into the formation,
138
479080
3296
08:14
or how many robots you pull out,
139
482400
2576
08:17
each robot is simply
reacting to its neighbor.
reacting to its neighbor.
140
485000
3136
08:20
It's aware of the fact that it needs
to form the circular shape,
to form the circular shape,
141
488160
4976
08:25
but collaborating with its neighbors
142
493160
1776
08:26
it forms the shape
without central coordination.
without central coordination.
143
494960
3720
08:31
Now if you put these ideas together,
144
499520
2416
08:33
the third idea is that we
essentially give these robots
essentially give these robots
145
501960
3896
08:37
mathematical descriptions
of the shape they need to execute.
of the shape they need to execute.
146
505880
4296
08:42
And these shapes can be varying
as a function of time,
as a function of time,
147
510200
3496
08:45
and you'll see these robots
start from a circular formation,
start from a circular formation,
148
513720
4496
08:50
change into a rectangular formation,
stretch into a straight line,
stretch into a straight line,
149
518240
3256
08:53
back into an ellipse.
150
521520
1375
08:54
And they do this with the same
kind of split-second coordination
kind of split-second coordination
151
522919
3617
08:58
that you see in natural swarms, in nature.
152
526560
3280
09:03
So why work with swarms?
153
531080
2136
09:05
Let me tell you about two applications
that we are very interested in.
that we are very interested in.
154
533240
4120
09:10
The first one has to do with agriculture,
155
538160
2376
09:12
which is probably the biggest problem
that we're facing worldwide.
that we're facing worldwide.
156
540560
3360
09:16
As you well know,
157
544760
1256
09:18
one in every seven persons
in this earth is malnourished.
in this earth is malnourished.
158
546040
3520
09:21
Most of the land that we can cultivate
has already been cultivated.
has already been cultivated.
159
549920
3480
09:25
And the efficiency of most systems
in the world is improving,
in the world is improving,
160
553960
3216
09:29
but our production system
efficiency is actually declining.
efficiency is actually declining.
161
557200
3520
09:33
And that's mostly because of water
shortage, crop diseases, climate change
shortage, crop diseases, climate change
162
561080
4216
09:37
and a couple of other things.
163
565320
1520
09:39
So what can robots do?
164
567360
1480
09:41
Well, we adopt an approach that's
called Precision Farming in the community.
called Precision Farming in the community.
165
569200
4616
09:45
And the basic idea is that we fly
aerial robots through orchards,
aerial robots through orchards,
166
573840
5376
09:51
and then we build
precision models of individual plants.
precision models of individual plants.
167
579240
3120
09:54
So just like personalized medicine,
168
582829
1667
09:56
while you might imagine wanting
to treat every patient individually,
to treat every patient individually,
169
584520
4816
10:01
what we'd like to do is build
models of individual plants
models of individual plants
170
589360
3696
10:05
and then tell the farmer
what kind of inputs every plant needs --
what kind of inputs every plant needs --
171
593080
4136
10:09
the inputs in this case being water,
fertilizer and pesticide.
fertilizer and pesticide.
172
597240
4440
10:14
Here you'll see robots
traveling through an apple orchard,
traveling through an apple orchard,
173
602640
3616
10:18
and in a minute you'll see
two of its companions
two of its companions
174
606280
2256
10:20
doing the same thing on the left side.
175
608560
1810
10:22
And what they're doing is essentially
building a map of the orchard.
building a map of the orchard.
176
610800
3656
10:26
Within the map is a map
of every plant in this orchard.
of every plant in this orchard.
177
614480
2816
10:29
(Robot buzzing)
178
617320
1656
10:31
Let's see what those maps look like.
179
619000
1896
10:32
In the next video, you'll see the cameras
that are being used on this robot.
that are being used on this robot.
180
620920
4296
10:37
On the top-left is essentially
a standard color camera.
a standard color camera.
181
625240
3240
10:41
On the left-center is an infrared camera.
182
629640
3296
10:44
And on the bottom-left
is a thermal camera.
is a thermal camera.
183
632960
3776
10:48
And on the main panel, you're seeing
a three-dimensional reconstruction
a three-dimensional reconstruction
184
636760
3336
10:52
of every tree in the orchard
as the sensors fly right past the trees.
as the sensors fly right past the trees.
185
640120
6120
10:59
Armed with information like this,
we can do several things.
we can do several things.
186
647640
4040
11:04
The first and possibly the most important
thing we can do is very simple:
thing we can do is very simple:
187
652200
4256
11:08
count the number of fruits on every tree.
188
656480
2440
11:11
By doing this, you tell the farmer
how many fruits she has in every tree
how many fruits she has in every tree
189
659520
4536
11:16
and allow her to estimate
the yield in the orchard,
the yield in the orchard,
190
664080
4256
11:20
optimizing the production
chain downstream.
chain downstream.
191
668360
2840
11:23
The second thing we can do
192
671640
1616
11:25
is take models of plants, construct
three-dimensional reconstructions,
three-dimensional reconstructions,
193
673280
4496
11:29
and from that estimate the canopy size,
194
677800
2536
11:32
and then correlate the canopy size
to the amount of leaf area on every plant.
to the amount of leaf area on every plant.
195
680360
3776
11:36
And this is called the leaf area index.
196
684160
2176
11:38
So if you know this leaf area index,
197
686360
1936
11:40
you essentially have a measure of how much
photosynthesis is possible in every plant,
photosynthesis is possible in every plant,
198
688320
5456
11:45
which again tells you
how healthy each plant is.
how healthy each plant is.
199
693800
2880
11:49
By combining visual
and infrared information,
and infrared information,
200
697520
4216
11:53
we can also compute indices such as NDVI.
201
701760
3296
11:57
And in this particular case,
you can essentially see
you can essentially see
202
705080
2816
11:59
there are some crops that are
not doing as well as other crops.
not doing as well as other crops.
203
707920
3016
12:02
This is easily discernible from imagery,
204
710960
4056
12:07
not just visual imagery but combining
205
715040
2216
12:09
both visual imagery and infrared imagery.
206
717280
2776
12:12
And then lastly,
207
720080
1336
12:13
one thing we're interested in doing is
detecting the early onset of chlorosis --
detecting the early onset of chlorosis --
208
721440
4016
12:17
and this is an orange tree --
209
725480
1496
12:19
which is essentially seen
by yellowing of leaves.
by yellowing of leaves.
210
727000
2560
12:21
But robots flying overhead
can easily spot this autonomously
can easily spot this autonomously
211
729880
3896
12:25
and then report to the farmer
that he or she has a problem
that he or she has a problem
212
733800
2936
12:28
in this section of the orchard.
213
736760
1520
12:30
Systems like this can really help,
214
738800
2696
12:33
and we're projecting yields
that can improve by about ten percent
that can improve by about ten percent
215
741520
5816
12:39
and, more importantly, decrease
the amount of inputs such as water
the amount of inputs such as water
216
747360
3216
12:42
by 25 percent by using
aerial robot swarms.
aerial robot swarms.
217
750600
3280
12:47
Lastly, I want you to applaud
the people who actually create the future,
the people who actually create the future,
218
755200
5736
12:52
Yash Mulgaonkar, Sikang Liu
and Giuseppe Loianno,
and Giuseppe Loianno,
219
760960
4920
12:57
who are responsible for the three
demonstrations that you saw.
demonstrations that you saw.
220
765920
3496
13:01
Thank you.
221
769440
1176
13:02
(Applause)
222
770640
5920
ABOUT THE SPEAKER
Vijay Kumar - RoboticistAs the dean of the University of Pennsylvania's School of Engineering and Applied Science, Vijay Kumar studies the control and coordination of multi-robot formations.
Why you should listen
At the General Robotics, Automation, Sensing and Perception (GRASP) Lab at the University of Pennsylvania, flying quadrotor robots move together in eerie formation, tightening themselves into perfect battalions, even filling in the gap when one of their own drops out. You might have seen viral videos of the quads zipping around the netting-draped GRASP Lab (they juggle! they fly through a hula hoop!). Vijay Kumar headed this lab from 1998-2004. He's now the dean of the School of Engineering and Applied Science at the University of Pennsylvania in Philadelphia, where he continues his work in robotics, blending computer science and mechanical engineering to create the next generation of robotic wonders.
Vijay Kumar | Speaker | TED.com