TEDIndia 2009

Pranav Mistry: The thrilling potential of SixthSense technology

Filmed:

At TEDIndia, Pranav Mistry demos several tools that help the physical world interact with the world of data -- including a deep look at his SixthSense device and a new, paradigm-shifting paper "laptop." In an onstage Q&A, Mistry says he'll open-source the software behind SixthSense, to open its possibilities to all.

- Director of research, Samsung Research America
As an MIT grad student, Pranav Mistry invented SixthSense, a wearable device that enables new interactions between the real world and the world of data. Full bio

We grew up
00:15
interacting with the physical objects around us.
00:17
There are an enormous number of them
00:20
that we use every day.
00:22
Unlike most of our computing devices,
00:24
these objects are much more fun to use.
00:27
When you talk about objects,
00:30
one other thing automatically comes attached to that thing,
00:33
and that is gestures:
00:36
how we manipulate these objects,
00:38
how we use these objects in everyday life.
00:40
We use gestures not only to interact with these objects,
00:43
but we also use them to interact with each other.
00:46
A gesture of "Namaste!", maybe, to respect someone,
00:48
or maybe --
00:51
in India I don't need to teach a kid that this means
00:52
"four runs" in cricket.
00:54
It comes as a part of our everyday learning.
00:56
So, I am very interested,
00:59
from the beginning, that how --
01:01
how our knowledge
01:03
about everyday objects and gestures,
01:05
and how we use these objects,
01:07
can be leveraged to our interactions with the digital world.
01:09
Rather than using a keyboard and mouse,
01:12
why can I not use my computer
01:15
in the same way that I interact in the physical world?
01:18
So, I started this exploration around eight years back,
01:21
and it literally started with a mouse on my desk.
01:24
Rather than using it for my computer,
01:27
I actually opened it.
01:30
Most of you might be aware that, in those days,
01:33
the mouse used to come with a ball inside,
01:35
and there were two rollers
01:37
that actually guide the computer where the ball is moving,
01:39
and, accordingly, where the mouse is moving.
01:42
So, I was interested in these two rollers,
01:44
and I actually wanted more, so I borrowed another mouse from a friend --
01:47
never returned to him --
01:50
and I now had four rollers.
01:52
Interestingly, what I did with these rollers is,
01:54
basically, I took them off of these mouses
01:57
and then put them in one line.
02:00
It had some strings and pulleys and some springs.
02:02
What I got is basically a gesture interface device
02:05
that actually acts as a motion-sensing device
02:08
made for two dollars.
02:12
So, here, whatever movement I do in my physical world
02:14
is actually replicated inside the digital world
02:17
just using this small device that I made, around eight years back,
02:20
in 2000.
02:23
Because I was interested in integrating these two worlds,
02:25
I thought of sticky notes.
02:27
I thought, "Why can I not connect
02:29
the normal interface of a physical sticky note
02:32
to the digital world?"
02:34
A message written on a sticky note to my mom
02:36
on paper
02:38
can come to an SMS,
02:39
or maybe a meeting reminder
02:41
automatically syncs with my digital calendar --
02:43
a to-do list that automatically syncs with you.
02:45
But you can also search in the digital world,
02:48
or maybe you can write a query, saying,
02:51
"What is Dr. Smith's address?"
02:53
and this small system actually prints it out --
02:55
so it actually acts like a paper input-output system,
02:57
just made out of paper.
02:59
In another exploration,
03:05
I thought of making a pen that can draw in three dimensions.
03:07
So, I implemented this pen
03:10
that can help designers and architects
03:12
not only think in three dimensions,
03:14
but they can actually draw
03:16
so that it's more intuitive to use that way.
03:18
Then I thought, "Why not make a Google Map,
03:20
but in the physical world?"
03:22
Rather than typing a keyword to find something,
03:24
I put my objects on top of it.
03:27
If I put a boarding pass, it will show me where the flight gate is.
03:29
A coffee cup will show where you can find more coffee,
03:32
or where you can trash the cup.
03:35
So, these were some of the earlier explorations I did because
03:37
the goal was to connect these two worlds seamlessly.
03:40
Among all these experiments,
03:44
there was one thing in common:
03:46
I was trying to bring a part of the physical world to the digital world.
03:48
I was taking some part of the objects,
03:52
or any of the intuitiveness of real life,
03:55
and bringing them to the digital world,
03:58
because the goal was to make our computing interfaces more intuitive.
04:01
But then I realized that we humans
04:04
are not actually interested in computing.
04:06
What we are interested in is information.
04:09
We want to know about things.
04:12
We want to know about dynamic things going around.
04:14
So I thought, around last year -- in the beginning of the last year --
04:16
I started thinking, "Why can I not take this approach in the reverse way?"
04:21
Maybe, "How about I take my digital world
04:24
and paint the physical world with that digital information?"
04:27
Because pixels are actually, right now, confined in these rectangular devices
04:32
that fit in our pockets.
04:36
Why can I not remove this confine
04:38
and take that to my everyday objects, everyday life
04:41
so that I don't need to learn the new language
04:44
for interacting with those pixels?
04:46
So, in order to realize this dream,
04:50
I actually thought of putting a big-size projector on my head.
04:52
I think that's why this is called a head-mounted projector, isn't it?
04:55
I took it very literally,
04:58
and took my bike helmet,
05:00
put a little cut over there so that the projector actually fits nicely.
05:02
So now, what I can do --
05:05
I can augment the world around me with this digital information.
05:07
But later,
05:11
I realized that I actually wanted to interact with those digital pixels, also.
05:13
So I put a small camera over there,
05:16
that acts as a digital eye.
05:18
Later, we moved to a much better,
05:20
consumer-oriented pendant version of that,
05:22
that many of you now know as the SixthSense device.
05:24
But the most interesting thing about this particular technology
05:27
is that you can carry your digital world with you
05:30
wherever you go.
05:34
You can start using any surface, any wall around you,
05:36
as an interface.
05:39
The camera is actually tracking all your gestures.
05:41
Whatever you're doing with your hands,
05:44
it's understanding that gesture.
05:46
And, actually, if you see, there are some color markers
05:48
that in the beginning version we are using with it.
05:50
You can start painting on any wall.
05:53
You stop by a wall, and start painting on that wall.
05:55
But we are not only tracking one finger, here.
05:58
We are giving you the freedom of using all of both of your hands,
06:00
so you can actually use both of your hands to zoom into or zoom out
06:04
of a map just by pinching all present.
06:07
The camera is actually doing --
06:09
just, getting all the images --
06:12
is doing the edge recognition and also the color recognition
06:13
and so many other small algorithms are going on inside.
06:16
So, technically, it's a little bit complex,
06:19
but it gives you an output which is more intuitive to use, in some sense.
06:21
But I'm more excited that you can actually take it outside.
06:24
Rather than getting your camera out of your pocket,
06:27
you can just do the gesture of taking a photo
06:30
and it takes a photo for you.
06:33
(Applause)
06:35
Thank you.
06:39
And later I can find a wall, anywhere,
06:41
and start browsing those photos
06:43
or maybe, "OK, I want to modify this photo a little bit
06:45
and send it as an email to a friend."
06:47
So, we are looking for an era where
06:49
computing will actually merge with the physical world.
06:52
And, of course, if you don't have any surface,
06:55
you can start using your palm for simple operations.
06:58
Here, I'm dialing a phone number just using my hand.
07:01
The camera is actually not only understanding your hand movements,
07:07
but, interestingly,
07:10
is also able to understand what objects you are holding in your hand.
07:11
What we're doing here is actually --
07:14
for example, in this case,
07:17
the book cover is matched
07:19
with so many thousands, or maybe millions of books online,
07:21
and checking out which book it is.
07:24
Once it has that information,
07:26
it finds out more reviews about that,
07:27
or maybe New York Times has a sound overview on that,
07:29
so you can actually hear, on a physical book,
07:32
a review as sound.
07:34
("famous talk at Harvard University ...")
07:36
This was Obama's visit last week to MIT.
07:38
("... and particularly I want to thank two outstanding MIT ...")
07:42
So, I was seeing the live [video] of his talk, outside, on just a newspaper.
07:46
Your newspaper will show you live weather information
07:51
rather than having it updated -- like, you have to check your computer
07:54
in order to do that, right?
07:57
(Applause)
07:59
When I'm going back, I can just use my boarding pass
08:04
to check how much my flight has been delayed,
08:07
because at that particular time,
08:09
I'm not feeling like opening my iPhone,
08:11
and checking out a particular icon.
08:13
And I think this technology will not only change the way --
08:15
yes. (Laughter)
08:18
It will change the way we interact with people, also,
08:20
not only the physical world.
08:22
The fun part is, I'm going to the Boston metro,
08:24
and playing a pong game inside the train
08:27
on the ground, right?
08:30
(Laughter)
08:32
And I think the imagination is the only limit
08:33
of what you can think of
08:35
when this kind of technology merges with real life.
08:37
But many of you argue, actually, that
08:39
all of our work is not only about physical objects.
08:41
We actually do lots of accounting and paper editing
08:44
and all those kinds of things; what about that?
08:47
And many of you are excited about the next generation tablet computers
08:49
to come out in the market.
08:53
So, rather than waiting for that,
08:55
I actually made my own, just using a piece of paper.
08:57
So, what I did here is remove the camera --
09:00
All the webcam cameras have a microphone inside the camera.
09:02
I removed the microphone from that,
09:06
and then just pinched that --
09:09
like I just made a clip out of the microphone --
09:11
and clipped that to a piece of paper, any paper that you found around.
09:14
So now the sound of the touch
09:18
is getting me when exactly I'm touching the paper.
09:21
But the camera is actually tracking where my fingers are moving.
09:24
You can of course watch movies.
09:28
("Good afternoon. My name is Russell ...
09:31
and I am a Wilderness Explorer in Tribe 54.")
09:34
And you can of course play games.
09:37
(Car engine)
09:40
Here, the camera is actually understanding how you're holding the paper
09:43
and playing a car-racing game.
09:46
(Applause)
09:48
Many of you already must have thought, OK, you can browse.
09:52
Yeah. Of course you can browse to any websites
09:54
or you can do all sorts of computing on a piece of paper
09:57
wherever you need it.
10:00
So, more interestingly,
10:01
I'm interested in how we can take that in a more dynamic way.
10:04
When I come back to my desk I can just pinch that information
10:07
back to my desktop
10:10
so I can use my full-size computer.
10:12
(Applause)
10:15
And why only computers? We can just play with papers.
10:17
Paper world is interesting to play with.
10:20
Here, I'm taking a part of a document
10:23
and putting over here a second part from a second place --
10:25
and I'm actually modifying the information
10:29
that I have over there.
10:32
Yeah. And I say, "OK, this looks nice,
10:34
let me print it out, that thing."
10:37
So I now have a print-out of that thing, and now --
10:39
the workflow is more intuitive the way we used to do it
10:41
maybe 20 years back,
10:44
rather than now switching between these two worlds.
10:47
So, as a last thought,
10:50
I think that integrating information to everyday objects
10:53
will not only help us to get rid of the digital divide,
10:56
the gap between these two worlds,
11:01
but will also help us, in some way,
11:03
to stay human,
11:05
to be more connected to our physical world.
11:07
And it will actually help us not end up being machines
11:13
sitting in front of other machines.
11:16
That's all. Thank you.
11:18
(Applause)
11:21
Thank you.
11:35
(Applause)
11:36
Chris Anderson: So, Pranav,
11:39
first of all, you're a genius.
11:40
This is incredible, really.
11:43
What are you doing with this? Is there a company being planned?
11:46
Or is this research forever, or what?
11:49
Pranav Mistry: So, there are lots of companies --
11:51
actually sponsor companies of Media Lab --
11:53
interested in taking this ahead in one or another way.
11:54
Companies like mobile phone operators
11:57
want to take this in a different way than the NGOs in India,
11:59
[who] are thinking, "Why can we only have 'Sixth Sense'?
12:02
We should have a 'Fifth Sense' for missing-sense people
12:05
who cannot speak.
12:07
This technology can be used for them to speak out in a different way
12:08
with maybe a speaker system."
12:11
CA: What are your own plans? Are you staying at MIT,
12:12
or are you going to do something with this?
12:15
PM: I'm trying to make this more available to people
12:16
so that anyone can develop their own SixthSense device,
12:18
because the hardware is actually not that hard to manufacture
12:21
or hard to make your own.
12:26
We will provide all the open source software for them,
12:28
maybe starting next month.
12:30
CA: Open source? Wow.
12:32
(Applause)
12:34
CA: Are you going to come back to India with some of this, at some point?
12:39
PM: Yeah. Yes, yes, of course.
12:42
CA: What are your plans? MIT?
12:44
India? How are you going to split your time going forward?
12:46
PM: There is a lot of energy here. Lots of learning.
12:48
All of this work that you have seen is all about
12:51
my learning in India.
12:53
And now, if you see, it's more about the cost-effectiveness:
12:55
this system costs you $300
12:58
compared to the $20,000 surface tables, or anything like that.
13:00
Or maybe even the $2 mouse gesture system
13:03
at that time was costing around $5,000?
13:06
So, we actually -- I showed that, at a conference, to
13:09
President Abdul Kalam, at that time,
13:13
and then he said, "OK, we should use this in Bhabha Atomic Research Centre
13:15
for some use of that."
13:18
So I'm excited about how I can bring the technology to the masses
13:20
rather than just keeping that technology in the lab environment.
13:23
(Applause)
13:26
CA: Based on the people we've seen at TED,
13:30
I would say you're truly one of the two or three
13:33
best inventors in the world right now.
13:34
It's an honor to have you at TED.
13:36
Thank you so much.
13:38
That's fantastic.
13:40
(Applause)
13:41

▲Back to top

About the Speaker:

Pranav Mistry - Director of research, Samsung Research America
As an MIT grad student, Pranav Mistry invented SixthSense, a wearable device that enables new interactions between the real world and the world of data.

Why you should listen

When Pranav Mistry was a PhD student in the Fluid Interfaces Group at MIT's Media Lab, he worked with lab director Pattie Maes to create some of the most entertaining and thought-provoking interfaces the world had ever seen. And not just computer interfaces, mind you -- these are ways to help the digital and the actual worlds interface. Imagine: intelligent sticky notes, Quickies, that can be searched and can send reminders; a pen that draws in 3D; and TaPuMa, a tangible public map that can act as Google of physical world. And of course the legendary SixthSense, which is now open sourced

Before his studies at MIT, he worked with Microsoft as a UX researcher; he's a graduate of IIT. Now, as director of research at Samsung Research America, Mistry heads the Think Tank Team, an interdisciplinary group of researchers that hunts for new ways to mix digital informational with real-world interactions. As an example, Mistry launched the company's smartwatch, the Galaxy Gear, in 2013.

More profile about the speaker
Pranav Mistry | Speaker | TED.com