08:43
TED2009

Pattie Maes + Pranav Mistry: Meet the SixthSense interaction

Filmed:

This demo -- from Pattie Maes' lab at MIT, spearheaded by Pranav Mistry -- was the buzz of TED. It's a wearable device with a projector that paves the way for profound interaction with our environment. Imagine "Minority Report" and then some.

- Researcher
As head of the MIT Media Lab's Fluid Interfaces Group, Pattie Maes researches the tools we use to work with information and connect with one another. Full bio

- Director of research, Samsung Research America
As an MIT grad student, Pranav Mistry invented SixthSense, a wearable device that enables new interactions between the real world and the world of data. Full bio

I've been intrigued by this question
00:12
of whether we could evolve or develop a sixth sense --
00:14
a sense that would give us seamless access
00:17
and easy access to meta-information
00:23
or information that may exist somewhere
00:26
that may be relevant to help us make the right decision
00:29
about whatever it is that we're coming across.
00:32
And some of you may argue,
00:35
well, don't today's cell phones do that already?
00:38
But I would say no.
00:41
When you meet someone here at TED --
00:43
and this is the top networking place, of course, of the year --
00:45
you don't shake somebody's hand
00:48
and then say, "Can you hold on for a moment
00:50
while I take out my phone and Google you?"
00:53
Or when you go to the supermarket
00:55
and you're standing there in that huge aisle
00:59
of different types of toilet papers,
01:01
you don't take out your cell phone, and open a browser,
01:04
and go to a website to try to decide
01:08
which of these different toilet papers
01:10
is the most ecologically responsible purchase to make.
01:13
So we don't really have easy access
01:16
to all this relevant information
01:19
that can just help us make optimal decisions
01:21
about what to do next and what actions to take.
01:24
And so my research group at the Media Lab
01:27
has been developing a series of inventions
01:31
to give us access to this information
01:35
in a sort of easy way,
01:38
without requiring that the user changes any of their behavior.
01:40
And I'm here to unveil
01:45
our latest effort,
01:47
and most successful effort so far,
01:50
which is still very much a work in process.
01:52
I'm actually wearing the device right now
01:54
and we've sort of cobbled it together
01:57
with components that are off the shelf --
02:00
and that, by the way, only cost 350 dollars
02:03
at this point in time.
02:06
I'm wearing a camera, just a simple webcam,
02:08
a portable, battery-powered projection system with a little mirror.
02:12
These components communicate to my cell phone in my pocket
02:18
which acts as the communication and computation device.
02:22
And in the video here we see my student Pranav Mistry,
02:26
who's really the genius who's been implementing
02:30
and designing this whole system.
02:33
And we see how this system
02:35
lets him walk up to any surface
02:37
and start using his hands to interact with the information
02:41
that is projected in front of him.
02:45
The system tracks the four significant fingers.
02:47
In this case, he's wearing simple marker caps
02:51
that you may recognize.
02:55
But if you want a more stylish version
02:57
you could also paint your nails in different colors.
02:59
And the camera basically tracks these four fingers
03:03
and recognizes any gestures that he's making
03:07
so he can just go to, for example, a map of Long Beach,
03:10
zoom in and out, etc.
03:15
The system also recognizes iconic gestures
03:17
such as the "take a picture" gesture,
03:20
and then takes a picture of whatever is in front of you.
03:23
And when he then walks back to the Media Lab,
03:26
he can just go up to any wall
03:30
and project all the pictures that he's taken,
03:32
sort through them and organize them,
03:35
and re-size them, etc.,
03:37
again using all natural gestures.
03:39
So, some of you most likely were here two years ago
03:42
and saw the demo by Jeff Han
03:46
or some of you may think, "Well, doesn't this look like the Microsoft Surface Table?"
03:51
And yes, you also interact using natural gestures,
03:54
both hands, etc.
03:59
But the difference here is that you can use any surface,
04:01
you can walk to up to any surface,
04:04
including your hand if nothing else is available
04:07
and interact with this projected data.
04:10
The device is completely portable,
04:13
and can be ...
04:16
(Applause)
04:18
So one important difference is that it's totally mobile.
04:25
Another even more important difference is that in mass production
04:29
this would not cost more tomorrow than today's cell phones
04:33
and would actually not sort of be a bigger packaging --
04:37
could look a lot more stylish
04:41
than this version that I'm wearing around my neck.
04:43
But other than letting some of you live out your fantasy
04:46
of looking as cool as Tom Cruise in "Minority Report,"
04:52
the reason why we're really excited about this device
04:56
is that it really can act as one of these sixth-sense devices
05:00
that gives you relevant information
05:06
about whatever is in front of you.
05:09
So we see Pranav here going into the supermarket
05:11
and he's shopping for some paper towels.
05:16
And, as he picks up a product the system can recognize
05:19
the product that he's picking up,
05:22
using either image recognition or marker technology,
05:24
and give him the green light or an orange light.
05:28
He can ask for additional information.
05:31
So this particular choice here
05:34
is a particularly good choice, given his personal criteria.
05:38
Some of you may want the toilet paper with the most bleach in it
05:42
rather than the most ecologically-responsible choice.
05:46
(Laughter)
05:49
If he picks up a book in the bookstore,
05:52
he can get an Amazon rating --
05:55
it gets projected right on the cover of the book.
05:57
This is Juan's book, our previous speaker,
05:59
which gets a great rating, by the way, at Amazon.
06:04
And so, Pranav turns the page of the book
06:06
and can then see additional information about the book --
06:09
reader comments, maybe sort of information by his favorite critic, etc.
06:12
If he turns to a particular page
06:19
he finds an annotation by maybe an expert of a friend of ours
06:21
that gives him a little bit of additional information
06:25
about whatever is on that particular page.
06:27
Reading the newspaper --
06:30
it never has to be outdated.
06:32
(Laughter)
06:35
You can get video annotations of the event that you're reading about
06:37
You can get the latest sports scores etc.
06:41
This is a more controversial one.
06:45
(Laughter)
06:48
As you interact with someone at TED,
06:49
maybe you can see a word cloud of the tags,
06:52
the words that are associated with that person
06:56
in their blog and personal web pages.
06:58
In this case, the student is interested in cameras, etc.
07:01
On your way to the airport,
07:06
if you pick up your boarding pass, it can tell you that your flight is delayed,
07:08
that the gate has changed, etc.
07:12
And, if you need to know what the current time is
07:15
it's as simple as drawing a watch --
07:18
(Laughter)
07:22
(Applause)
07:23
on your arm.
07:24
So that's where we're at so far
07:26
in developing this sixth sense
07:30
that would give us seamless access to all this relevant information
07:34
about the things that we may come across.
07:38
My student Pranav, who's really, like I said, the genius behind this.
07:41
(Applause) (Standing ovation)
07:45
He does deserve a lot of applause
08:08
because I don't think he's slept much in the last three months, actually.
08:11
And his girlfriend is probably not very happy about him either.
08:15
But it's not perfect yet, it's very much a work in progress.
08:18
And who knows, maybe in another 10 years
08:23
we'll be here with the ultimate sixth sense brain implant.
08:26
Thank you.
08:31
(Applause)
08:32

▲Back to top

About the Speakers:

Pattie Maes - Researcher
As head of the MIT Media Lab's Fluid Interfaces Group, Pattie Maes researches the tools we use to work with information and connect with one another.

Why you should listen

Pattie Maes was the key architect behind what was once called "collaborative filtering" and has become a key to Web 2.0: the immense engine of recommendations -- or "things like this" -- fueled by other users. In the 1990s, Maes' Software Agents program at MIT created Firefly, a technology (and then a startup sold to Microsoft) that let users choose songs they liked, and find similar songs they'd never heard of, by taking cues from others with similar taste. This brought a sea change in the way we interact with software, with culture and with one another.

Now Maes is working on a similarly boundary-breaking initiative. She founded Fluid Interfaces Group, also part of the MIT Media Lab, to rethink the ways in which humans and computers interact, partially by redefining both human and computer. In Maes' world (and really, in all of ours), the computer is no longer a distinct object, but a source of intelligence that's embedded in our environment. By outfitting ourselves with digital accessories, we can continually learn from (and teach) our surroundings. The uses of this tech -- from healthcare to home furnishings, warfare to supermarkets -- are powerful and increasingly real.

More profile about the speaker
Pattie Maes | Speaker | TED.com
Pranav Mistry - Director of research, Samsung Research America
As an MIT grad student, Pranav Mistry invented SixthSense, a wearable device that enables new interactions between the real world and the world of data.

Why you should listen

When Pranav Mistry was a PhD student in the Fluid Interfaces Group at MIT's Media Lab, he worked with lab director Pattie Maes to create some of the most entertaining and thought-provoking interfaces the world had ever seen. And not just computer interfaces, mind you -- these are ways to help the digital and the actual worlds interface. Imagine: intelligent sticky notes, Quickies, that can be searched and can send reminders; a pen that draws in 3D; and TaPuMa, a tangible public map that can act as Google of physical world. And of course the legendary SixthSense, which is now open sourced

Before his studies at MIT, he worked with Microsoft as a UX researcher; he's a graduate of IIT. Now, as director of research at Samsung Research America, Mistry heads the Think Tank Team, an interdisciplinary group of researchers that hunts for new ways to mix digital informational with real-world interactions. As an example, Mistry launched the company's smartwatch, the Galaxy Gear, in 2013.

More profile about the speaker
Pranav Mistry | Speaker | TED.com