TED2010
John Underkoffler: Pointing to the future of UI
Filmed:
Readability: 4
1,888,233 views
Minority Report science adviser and inventor John Underkoffler demos g-speak -- the real-life version of the film's eye-popping, tai chi-meets-cyberspace computer interface. Is this how tomorrow's computers will be controlled?
John Underkoffler - Interface designer
Remember the data interface from Minority Report? Well, it's real, John Underkoffler invented it -- as a point-and-touch interface called g-speak -- and it's about to change the way we interact with data. Full bio
Remember the data interface from Minority Report? Well, it's real, John Underkoffler invented it -- as a point-and-touch interface called g-speak -- and it's about to change the way we interact with data. Full bio
Double-click the English transcript below to play the video.
00:15
We're 25, 26 years after
0
0
2000
00:17
the advent of the Macintosh,
1
2000
2000
00:19
which was an astoundingly seminal event
2
4000
2000
00:21
in the history
3
6000
2000
00:23
of human-machine interface
4
8000
2000
00:25
and in computation in general.
5
10000
2000
00:27
It fundamentally changed the way
6
12000
2000
00:29
that people thought about computation,
7
14000
2000
00:31
thought about computers,
8
16000
2000
00:33
how they used them and who and how many people were able to use them.
9
18000
3000
00:36
It was such a radical change, in fact,
10
21000
2000
00:38
that the early Macintosh development team
11
23000
2000
00:40
in '82, '83, '84
12
25000
2000
00:42
had to write an entirely new operating system from the ground up.
13
27000
3000
00:45
Now, this is an interesting little message,
14
30000
2000
00:47
and it's a lesson that has since, I think,
15
32000
2000
00:49
been forgotten or lost or something,
16
34000
2000
00:51
and that is, namely, that the OS is the interface.
17
36000
3000
00:54
The interface is the OS.
18
39000
2000
00:56
It's like the land and the king (i.e. Arthur) they're inseparable, they are one.
19
41000
3000
00:59
And to write a new operating system was not a capricious matter.
20
44000
3000
01:02
It wasn't just a matter of tuning up some graphics routines.
21
47000
3000
01:05
There were no graphics routines. There were no mouse drivers.
22
50000
3000
01:08
So it was a necessity.
23
53000
2000
01:10
But in the quarter-century since then,
24
55000
2000
01:12
we've seen all of the fundamental
25
57000
2000
01:14
supporting technologies go berserk.
26
59000
2000
01:16
So memory capacity and disk capacity
27
61000
3000
01:19
have been multiplied by something between 10,000 and a million.
28
64000
3000
01:22
Same thing for processor speeds.
29
67000
2000
01:24
Networks, we didn't have networks at all
30
69000
2000
01:26
at the time of the Macintosh's introduction,
31
71000
3000
01:29
and that has become the single most salient aspect
32
74000
2000
01:31
of how we live with computers.
33
76000
2000
01:33
And, of course, graphics: Today
34
78000
2000
01:35
84 dollars and 97 cents at Best Buy
35
80000
3000
01:38
buys you more graphics power
36
83000
2000
01:40
than you could have gotten for a million bucks from SGI only a decade ago.
37
85000
3000
01:43
So we've got that incredible ramp-up.
38
88000
2000
01:45
Then, on the side, we've got the Web
39
90000
2000
01:47
and, increasingly, the cloud,
40
92000
2000
01:49
which is fantastic,
41
94000
2000
01:51
but also -- in the regard in which an interface is fundamental --
42
96000
3000
01:54
kind of a distraction.
43
99000
2000
01:56
So we've forgotten to invent new interfaces.
44
101000
2000
01:58
Certainly we've seen in recent years a lot of change in that regard,
45
103000
2000
02:00
and people are starting to wake up about that.
46
105000
3000
02:04
So what happens next? Where do we go from there?
47
109000
2000
02:06
The problem, as we see it,
48
111000
2000
02:08
has to do with a single, simple word: "space,"
49
113000
2000
02:10
or a single, simple phrase:
50
115000
2000
02:12
"real world geometry."
51
117000
2000
02:14
Computers and the programming languages
52
119000
2000
02:16
that we talk to them in, that we teach them in,
53
121000
2000
02:18
are hideously insensate when it comes to space.
54
123000
3000
02:21
They don't understand real world space.
55
126000
2000
02:23
It's a funny thing because the rest of us occupy it quite frequently and quite well.
56
128000
3000
02:26
They also don't understand time, but that's a matter for a separate talk.
57
131000
3000
02:29
So what happens if you start to
58
134000
2000
02:31
explain space to them?
59
136000
2000
02:36
One thing you might get is something like the Luminous Room.
60
141000
3000
02:39
The Luminous Room is a system
61
144000
2000
02:41
in which it's considered that
62
146000
2000
02:43
input and output spaces are co-located.
63
148000
2000
02:45
That's a strangely simple,
64
150000
2000
02:47
and yet unexplored idea, right?
65
152000
2000
02:49
When you use a mouse, your hand is down here on the mouse pad.
66
154000
3000
02:52
It's not even on the same plane as what you're talking about:
67
157000
2000
02:54
The pixels are up on the display.
68
159000
2000
02:56
So here was a room in which all the walls, floors, ceilings,
69
161000
3000
02:59
pets, potted plants, whatever was in there,
70
164000
2000
03:01
were capable, not only of display but of sensing as well.
71
166000
3000
03:04
And that means input and output are in the same space
72
169000
2000
03:06
enabling stuff like this.
73
171000
2000
03:08
That's a digital storage in a physical container.
74
173000
2000
03:10
The contract is the same
75
175000
2000
03:12
as with real word objects in real world containers.
76
177000
3000
03:15
Has to come back out, whatever you put in.
77
180000
3000
03:18
This little design experiment
78
183000
2000
03:20
that was a small office here knew a few other tricks as well.
79
185000
3000
03:23
If you presented it with a chess board,
80
188000
2000
03:25
it tried to figure out what you might mean by that.
81
190000
2000
03:27
And if there was nothing for them to do,
82
192000
2000
03:29
the chess pieces eventually got bored
83
194000
2000
03:31
and hopped away.
84
196000
2000
03:33
The academics who were overseeing this work
85
198000
3000
03:36
thought that that was too frivolous,
86
201000
2000
03:38
so we built deadly serious applications
87
203000
2000
03:40
like this optics prototyping workbench
88
205000
2000
03:42
in which a toothpaste cap on a cardboard box
89
207000
3000
03:45
becomes a laser.
90
210000
2000
03:47
The beam splitters and lenses are represented by physical objects,
91
212000
3000
03:50
and the system projects down the laser beam path.
92
215000
3000
03:53
So you've got an interface that has no interface.
93
218000
2000
03:55
You operate the world as you operate the real world,
94
220000
3000
03:58
which is to say, with your hands.
95
223000
2000
04:00
Similarly, a digital wind tunnel with digital wind
96
225000
2000
04:02
flowing from right to left --
97
227000
2000
04:04
not that remarkable in a sense; we didn't invent the mathematics.
98
229000
3000
04:07
But if you displayed that on a CRT or flat panel display,
99
232000
2000
04:09
it would be meaningless to hold up an arbitrary object,
100
234000
3000
04:12
a real world object in that.
101
237000
2000
04:14
Here, the real world merges with the simulation.
102
239000
3000
04:18
And finally, to pull out all the stops,
103
243000
2000
04:20
this is a system called Urp, for urban planners,
104
245000
3000
04:23
in which we give architects and urban planners back
105
248000
3000
04:26
the models that we confiscated
106
251000
2000
04:28
when we insisted that they use CAD systems.
107
253000
2000
04:30
And we make the machine meet them half way.
108
255000
2000
04:32
It projects down digital shadows, as you see here.
109
257000
3000
04:35
And if you introduce tools like this inverse clock,
110
260000
3000
04:38
then you can control the sun's position in the sky.
111
263000
2000
04:40
That's 8 a.m. shadows.
112
265000
2000
04:42
They get a little shorter at 9 a.m.
113
267000
2000
04:44
There you are, swinging the sun around.
114
269000
2000
04:46
Short shadows at noon and so forth.
115
271000
3000
04:50
And we built up a series of tools like this.
116
275000
3000
04:53
There are inter-shadowing studies
117
278000
2000
04:55
that children can operate,
118
280000
2000
04:57
even though they don't know anything about urban planning:
119
282000
2000
04:59
To move a building, you simply reach out your hand and you move the building.
120
284000
3000
05:02
A material wand makes the building
121
287000
2000
05:04
into a sort of Frank Gehry thing that reflects light in all directions.
122
289000
3000
05:07
Are you blinding passers by and motorists on the freeways?
123
292000
3000
05:10
A zoning tool connects distant structures, a building and a roadway.
124
295000
3000
05:13
Are you going to get sued by the zoning commission? And so forth.
125
298000
3000
05:17
Now, if these ideas seem familiar
126
302000
2000
05:19
or perhaps even a little dated,
127
304000
2000
05:21
that's great; they should seem familiar.
128
306000
2000
05:23
This work is 15 years old.
129
308000
2000
05:26
This stuff was undertaken at MIT and the Media Lab
130
311000
3000
05:29
under the incredible direction of Professor Hiroshi Ishii,
131
314000
3000
05:32
director of the Tangible Media Group.
132
317000
2000
05:34
But it was that work that was seen
133
319000
2000
05:36
by Alex McDowell,
134
321000
2000
05:38
one of the world's legendary production designers.
135
323000
3000
05:41
But Alex was preparing a little, sort of obscure, indie, arthouse film
136
326000
3000
05:44
called "Minority Report" for Steven Spielberg,
137
329000
3000
05:47
and invited us to come out from MIT
138
332000
2000
05:49
and design the interfaces
139
334000
3000
05:52
that would appear in that film.
140
337000
3000
05:55
And the great thing about it was
141
340000
2000
05:57
that Alex was so dedicated to the idea of verisimilitude,
142
342000
3000
06:00
the idea that the putative 2054
143
345000
3000
06:03
that we were painting in the film be believable,
144
348000
3000
06:06
that he allowed us to take on that design work
145
351000
2000
06:08
as if it were an R&D effort.
146
353000
2000
06:10
And the result is sort of
147
355000
2000
06:12
gratifyingly perpetual.
148
357000
2000
06:14
People still reference those sequences in "Minority Report"
149
359000
3000
06:17
when they talk about new UI design.
150
362000
2000
06:19
So this led full circle, in a strange way,
151
364000
2000
06:21
to build these ideas into what we believe
152
366000
3000
06:24
is the necessary future of human machine interface:
153
369000
3000
06:27
the Spatial Operating Environment, we call it.
154
372000
3000
06:32
So here we have a bunch of stuff, some images.
155
377000
3000
06:35
And, using a hand,
156
380000
2000
06:37
we can actually exercise six degrees of freedom,
157
382000
3000
06:40
six degrees of navigational control.
158
385000
3000
06:43
And it's fun to fly through Mr. Beckett's eye.
159
388000
2000
06:45
And you can come back out
160
390000
2000
06:47
through the scary orangutan.
161
392000
2000
06:49
And that's all well and good.
162
394000
3000
06:52
Let's do something a little more difficult.
163
397000
3000
06:55
Here, we have a whole bunch of disparate images.
164
400000
2000
06:57
We can fly around them.
165
402000
2000
06:59
So navigation is a fundamental issue.
166
404000
2000
07:01
You have to be able to navigate in 3D.
167
406000
3000
07:04
Much of what we want computers to help us with in the first place
168
409000
3000
07:07
is inherently spatial.
169
412000
2000
07:09
And the part that isn't spatial can often be spatialized
170
414000
2000
07:11
to allow our wetware to make greater sense of it.
171
416000
3000
07:14
Now we can distribute this stuff in many different ways.
172
419000
3000
07:17
So we can throw it out like that. Let's reset it.
173
422000
2000
07:19
We can organize it this way.
174
424000
2000
07:21
And, of course, it's not just about navigation,
175
426000
3000
07:24
but about manipulation as well.
176
429000
2000
07:26
So if we don't like stuff,
177
431000
2000
07:28
or we're intensely curious about
178
433000
2000
07:30
Ernst Haeckel's scientific falsifications,
179
435000
3000
07:33
we can pull them out like that.
180
438000
2000
07:35
And then if it's time for analysis, we can pull back a little bit
181
440000
3000
07:38
and ask for a different distribution.
182
443000
3000
07:43
Let's just come down a bit
183
448000
2000
07:46
and fly around.
184
451000
2000
07:49
So that's a different way to look at stuff.
185
454000
3000
07:52
If you're of a more analytical nature
186
457000
2000
07:54
then you might want, actually, to look at this
187
459000
2000
07:56
as a color histogram.
188
461000
3000
07:59
So now we've got the stuff color-sorted,
189
464000
3000
08:02
angle maps onto color.
190
467000
3000
08:05
And now, if we want to select stuff,
191
470000
2000
08:07
3D, space,
192
472000
2000
08:09
the idea that we're tracking hands in real space
193
474000
3000
08:12
becomes really important because we can reach in,
194
477000
3000
08:15
not in 2D, not in fake 2D, but in actual 3D.
195
480000
2000
08:17
Here are some selection planes.
196
482000
2000
08:19
And we'll perform this Boolean operation
197
484000
3000
08:22
because we really love yellow and tapirs on green grass.
198
487000
3000
08:34
So, from there to the world of real work.
199
499000
3000
08:37
Here's a logistics system,
200
502000
2000
08:39
a small piece of one that we're currently building.
201
504000
2000
08:41
There're a lot of elements.
202
506000
2000
08:43
And one thing that's very important is to combine traditional tabular data
203
508000
3000
08:46
with three-dimensional and geospatial information.
204
511000
3000
08:49
So here's a familiar place.
205
514000
3000
08:52
And we'll bring this back here for a second.
206
517000
3000
08:55
Maybe select a little bit of that.
207
520000
3000
08:58
And bring out this graph.
208
523000
3000
09:01
And we should, now,
209
526000
2000
09:03
be able to fly in here
210
528000
3000
09:06
and have a closer look.
211
531000
3000
09:09
These are logistics elements
212
534000
2000
09:11
that are scattered across the United States.
213
536000
3000
09:20
One thing that three-dimensional interactions
214
545000
3000
09:23
and the general idea of imbuing
215
548000
2000
09:25
computation with space affords you
216
550000
2000
09:27
is a final destruction of that unfortunate
217
552000
2000
09:29
one-to-one pairing between human beings and computers.
218
554000
3000
09:32
That's the old way, that's the old mantra:
219
557000
2000
09:34
one machine, one human, one mouse, one screen.
220
559000
2000
09:36
Well, that doesn't really cut it anymore.
221
561000
3000
09:39
In the real world, we have people who collaborate;
222
564000
3000
09:42
we have people who have to work together,
223
567000
3000
09:45
and we have many different displays.
224
570000
3000
09:48
And we might want to look at these various images.
225
573000
3000
09:51
We might want to ask for some help.
226
576000
2000
09:53
The author of this new pointing device
227
578000
3000
09:56
is sitting over there,
228
581000
2000
09:58
so I can pull this from there to there.
229
583000
2000
10:00
These are unrelated machines, right?
230
585000
3000
10:03
So the computation is space soluble and network soluble.
231
588000
3000
10:06
So I'm going to leave that over there
232
591000
2000
10:08
because I have a question for Paul.
233
593000
2000
10:10
Paul is the designer of this wand, and maybe its easiest
234
595000
2000
10:12
for him to come over here and tell me in person what's going on.
235
597000
3000
10:15
So let me get some of these out of the way.
236
600000
3000
10:20
Let's pull this apart:
237
605000
2000
10:23
I'll go ahead and explode it.
238
608000
3000
10:26
Kevin, can you help?
239
611000
2000
10:33
Let me see if I can help us find the circuit board.
240
618000
3000
10:38
Mind you, it's a sort of gratuitous field-stripping exercise,
241
623000
3000
10:41
but we do it in the lab all the time.
242
626000
3000
10:44
All right.
243
629000
2000
10:46
So collaborative work, whether it's immediately co-located
244
631000
3000
10:49
or distant and distinct, is always important.
245
634000
3000
10:52
And again, that stuff
246
637000
2000
10:54
needs to be undertaken in the context of space.
247
639000
3000
10:59
And finally, I'd like to leave you with a glimpse
248
644000
3000
11:02
that takes us back to the world of imagery.
249
647000
2000
11:04
This is a system called TAMPER,
250
649000
2000
11:06
which is a slightly whimsical look
251
651000
2000
11:08
at what the future of editing
252
653000
2000
11:10
and media manipulation systems might be.
253
655000
2000
11:12
We at Oblong believe that media should be
254
657000
2000
11:14
accessible in much more fine-grained form.
255
659000
3000
11:17
So we have a large number of movies
256
662000
2000
11:19
stuck inside here.
257
664000
2000
11:21
And let's just pick out a few elements.
258
666000
3000
11:24
We can zip through them
259
669000
3000
11:27
as a possibility.
260
672000
2000
11:29
We can grab elements off the front,
261
674000
2000
11:31
where upon they reanimate, come to life,
262
676000
3000
11:34
and drag them down onto the table here.
263
679000
3000
11:39
We'll go over to Jacques Tati here
264
684000
3000
11:42
and grab our blue friend
265
687000
3000
11:45
and put him down on the table as well.
266
690000
3000
11:48
We may need more than one.
267
693000
3000
11:53
And we probably need,
268
698000
2000
11:55
well, we probably need a cowboy
269
700000
2000
11:57
to be quite honest.
270
702000
2000
11:59
(Laughter)
271
704000
2000
12:01
Yeah, let's
272
706000
3000
12:04
take that one.
273
709000
2000
12:06
(Laughter)
274
711000
2000
12:08
You see, cowboys and French farce people
275
713000
2000
12:10
don't go well together, and the system knows that.
276
715000
3000
12:15
Let me leave with one final thought,
277
720000
2000
12:17
and that is that
278
722000
2000
12:19
one of the greatest English language writers
279
724000
2000
12:21
of the last three decades
280
726000
2000
12:23
suggested that great art is always a gift.
281
728000
3000
12:26
And he wasn't talking about whether the novel costs 24.95 [dollars],
282
731000
3000
12:29
or whether you have to spring 70 million bucks
283
734000
2000
12:31
to buy the stolen Vermeer;
284
736000
2000
12:33
he was talking about the circumstances of its creation
285
738000
2000
12:35
and of its existence.
286
740000
2000
12:37
And I think that it's time that we asked
287
742000
2000
12:39
for the same from technology.
288
744000
2000
12:41
Technology is capable of
289
746000
2000
12:43
expressing and being imbued with
290
748000
3000
12:46
a certain generosity,
291
751000
2000
12:48
and we need to demand that, in fact.
292
753000
2000
12:50
For some of this kind of technology,
293
755000
3000
12:53
ground center is
294
758000
3000
12:56
a combination of design, which is crucially important.
295
761000
2000
12:58
We can't have advances in technology any longer
296
763000
3000
13:01
unless design is integrated from the very start.
297
766000
3000
13:04
And, as well, as of efficacy, agency.
298
769000
3000
13:07
We're, as human beings, the creatures that create,
299
772000
3000
13:10
and we should make sure that our machines aid us in that task
300
775000
3000
13:13
and are built in that same image.
301
778000
3000
13:16
So I will leave you with that. Thank you.
302
781000
2000
13:18
(Applause)
303
783000
13000
13:33
Chris Anderson: So to ask the obvious question --
304
798000
3000
13:36
actually this is from Bill Gates --
305
801000
2000
13:38
when? (John Underkoffler: When?)
306
803000
3000
13:41
CA: When real? When for us, not just in a lab and on a stage?
307
806000
3000
13:45
Can it be for every man, or is this just for corporations and movie producers?
308
810000
3000
13:48
JU: No, it has to be for every human being.
309
813000
2000
13:50
That's our goal entirely.
310
815000
2000
13:52
We won't have succeeded
311
817000
2000
13:54
unless we take that next big step.
312
819000
2000
13:56
I mean it's been 25 years.
313
821000
2000
13:58
Can there really be only one interface? There can't.
314
823000
2000
14:00
CA: But does that mean that, at your desk or in your home,
315
825000
2000
14:02
you need projectors, cameras?
316
827000
3000
14:05
You know, how can it work?
317
830000
2000
14:07
JU: No, this stuff will be built into the bezel of every display.
318
832000
2000
14:09
It'll be built into architecture.
319
834000
2000
14:11
The gloves go away in a matter of months or years.
320
836000
3000
14:14
So this is the inevitability about it.
321
839000
2000
14:16
CA: So, in your mind, five years time,
322
841000
2000
14:18
someone can buy this as part of
323
843000
2000
14:20
a standard computer interface?
324
845000
2000
14:22
JU: I think in five years time when you buy a computer,
325
847000
3000
14:25
you'll get this.
326
850000
2000
14:27
CA: Well that's cool.
327
852000
2000
14:29
(Applause)
328
854000
4000
14:33
The world has a habit of surprising us as to how these things are actually used.
329
858000
3000
14:36
What do you think, what in your mind is the first killer app for this?
330
861000
3000
14:39
JU: That's a good question, and we ask ourselves that every day.
331
864000
3000
14:42
At the moment, our early-adopter customers --
332
867000
3000
14:45
and these systems are deployed out in the real world --
333
870000
3000
14:48
do all the big data intensive, data heavy problems with it.
334
873000
3000
14:51
So, whether it's logistics and supply chain management
335
876000
2000
14:53
or natural gas and resource extraction,
336
878000
3000
14:56
financial services, pharmaceuticals, bioinformatics,
337
881000
3000
14:59
those are the topics right now, but that's not a killer app.
338
884000
2000
15:01
And I understand what you're asking.
339
886000
2000
15:03
CA: C'mon, c'mon. Martial arts, games. C'mon.
340
888000
2000
15:05
(Laughter)
341
890000
2000
15:07
John, thank you for making science-fiction real.
342
892000
3000
15:10
JU: It's been a great pleasure.
343
895000
2000
15:12
Thank you to you all.
344
897000
2000
15:14
(Applause)
345
899000
2000
ABOUT THE SPEAKER
John Underkoffler - Interface designerRemember the data interface from Minority Report? Well, it's real, John Underkoffler invented it -- as a point-and-touch interface called g-speak -- and it's about to change the way we interact with data.
Why you should listen
When Tom Cruise put on his data glove and started whooshing through video clips of future crimes, how many of us felt the stirrings of geek lust? This iconic scene in Minority Report marked a change in popular thinking about interfaces -- showing how sexy it could be to use natural gestures, without keyboard, mouse or command line.
John Underkoffler led the team that came up with this interface, called the g-speak Spatial Operating Environment. His company, Oblong Industries, was founded to move g-speak into the real world. Oblong is building apps for aerospace, bioinformatics, video editing and more. But the big vision is ubiquity: g-speak on every laptop, every desktop, every microwave oven, TV, dashboard. "It has to be like this," he says. "We all of us every day feel that. We build starting there. We want to change it all."
Before founding Oblong, Underkoffler spent 15 years at MIT's Media Laboratory, working in holography, animation and visualization techniques, and building the I/O Bulb and Luminous Room Systems.
John Underkoffler | Speaker | TED.com