sponsored links
TED2013

Ajit Narayanan: A word game to communicate in any language

February 27, 2013

While working with kids who have trouble speaking, Ajit Narayanan sketched out a way to think about language in pictures, to relate words and concepts in "maps." The idea now powers an app that helps nonverbal people communicate, and the big idea behind it, a language concept called FreeSpeech, has exciting potential.

Ajit Narayanan - Visual grammar engine inventor
Ajit Narayanan is the inventor of Avaz, an affordable, tablet-based communication device for people who are speech-impaired. Full bio

sponsored links
Double-click the English subtitles below to play the video.
I work with children with autism.
00:12
Specifically, I make technologies
00:15
to help them communicate.
00:17
Now, many of the problems that children
00:19
with autism face, they have a common source,
00:20
and that source is that they find it difficult
00:24
to understand abstraction, symbolism.
00:26
And because of this, they have
a lot of difficulty with language.
00:31
Let me tell you a little bit about why this is.
00:36
You see that this is a picture of a bowl of soup.
00:39
All of us can see it. All of us understand this.
00:43
These are two other pictures of soup,
00:46
but you can see that these are more abstract
00:48
These are not quite as concrete.
00:50
And when you get to language,
00:52
you see that it becomes a word
00:54
whose look, the way it looks and the way it sounds,
00:56
has absolutely nothing to do
with what it started with,
00:59
or what it represents, which is the bowl of soup.
01:02
So it's essentially a completely abstract,
01:05
a completely arbitrary representation of something
01:08
which is in the real world,
01:10
and this is something that children with autism
01:11
have an incredible amount of difficulty with.
01:13
Now that's why most of the people
that work with children with autism --
01:16
speech therapists, educators --
01:19
what they do is, they try to help children with autism
01:21
communicate not with words, but with pictures.
01:24
So if a child with autism wanted to say,
01:27
"I want soup," that child would pick
01:29
three different pictures, "I," "want," and "soup,"
01:31
and they would put these together,
01:34
and then the therapist or the parent would
01:35
understand that this is what the kid wants to say.
01:37
And this has been incredibly effective;
01:39
for the last 30, 40 years
01:41
people have been doing this.
01:43
In fact, a few years back,
01:44
I developed an app for the iPad
01:46
which does exactly this. It's called Avaz,
01:48
and the way it works is that kids select
01:51
different pictures.
01:53
These pictures are sequenced
together to form sentences,
01:54
and these sentences are spoken out.
01:57
So Avaz is essentially converting pictures,
01:59
it's a translator, it converts pictures into speech.
02:02
Now, this was very effective.
02:06
There are thousands of children using this,
02:07
you know, all over the world,
02:09
and I started thinking about
02:10
what it does and what it doesn't do.
02:12
And I realized something interesting:
02:15
Avaz helps children with autism learn words.
02:17
What it doesn't help them do is to learn
02:21
word patterns.
02:23
Let me explain this in a little more detail.
02:26
Take this sentence: "I want soup tonight."
02:28
Now it's not just the words
here that convey the meaning.
02:32
It's also the way in which these words are arranged,
02:36
the way these words are modified and arranged.
02:39
And that's why a sentence like "I want soup tonight"
02:41
is different from a sentence like
02:44
"Soup want I tonight," which
is completely meaningless.
02:46
So there is another hidden abstraction here
02:49
which children with autism find
a lot of difficulty coping with,
02:52
and that's the fact that you can modify words
02:55
and you can arrange them to have
02:58
different meanings, to convey different ideas.
03:00
Now, this is what we call grammar.
03:03
And grammar is incredibly powerful,
03:06
because grammar is this one component of language
03:08
which takes this finite vocabulary that all of us have
03:12
and allows us to convey an
infinite amount of information,
03:15
an infinite amount of ideas.
03:20
It's the way in which you can put things together
03:22
in order to convey anything you want to.
03:24
And so after I developed Avaz,
03:26
I worried for a very long time
03:28
about how I could give grammar
to children with autism.
03:30
The solution came to me from
a very interesting perspective.
03:33
I happened to chance upon a child with autism
03:36
conversing with her mom,
03:39
and this is what happened.
03:41
Completely out of the blue, very spontaneously,
03:43
the child got up and said, "Eat."
03:46
Now what was interesting was
03:48
the way in which the mom was trying to tease out
03:50
the meaning of what the child wanted to say
03:54
by talking to her in questions.
03:56
So she asked, "Eat what? Do
you want to eat ice cream?
03:59
You want to eat? Somebody else wants to eat?
04:01
You want to eat cream now? You
want to eat ice cream in the evening?"
04:03
And then it struck me that
04:07
what the mother had done was something incredible.
04:08
She had been able to get that child to communicate
04:10
an idea to her without grammar.
04:12
And it struck me that maybe this is what
04:16
I was looking for.
04:19
Instead of arranging words in an order, in sequence,
04:20
as a sentence, you arrange them
04:24
in this map, where they're all linked together
04:27
not by placing them one after the other
04:30
but in questions, in question-answer pairs.
04:33
And so if you do this, then what you're conveying
04:36
is not a sentence in English,
04:38
but what you're conveying is really a meaning,
04:40
the meaning of a sentence in English.
04:43
Now, meaning is really the underbelly,
in some sense, of language.
04:45
It's what comes after thought but before language.
04:48
And the idea was that this particular representation
04:51
might convey meaning in its raw form.
04:54
So I was very excited by this, you know,
04:57
hopping around all over the place,
04:59
trying to figure out if I can convert
05:00
all possible sentences that I hear into this.
05:02
And I found that this is not enough.
05:05
Why is this not enough?
05:07
This is not enough because if you wanted to convey
05:08
something like negation,
05:10
you want to say, "I don't want soup,"
05:12
then you can't do that by asking a question.
05:14
You do that by changing the word "want."
05:16
Again, if you wanted to say,
05:18
"I wanted soup yesterday,"
05:20
you do that by converting
the word "want" into "wanted."
05:22
It's a past tense.
05:24
So this is a flourish which I added
05:26
to make the system complete.
05:28
This is a map of words joined together
05:30
as questions and answers,
05:32
and with these filters applied on top of them
05:33
in order to modify them to represent
05:36
certain nuances.
05:38
Let me show you this with a different example.
05:39
Let's take this sentence:
05:41
"I told the carpenter I could not pay him."
05:42
It's a fairly complicated sentence.
05:44
The way that this particular system works,
05:46
you can start with any part of this sentence.
05:48
I'm going to start with the word "tell."
05:51
So this is the word "tell."
05:52
Now this happened in the past,
05:54
so I'm going to make that "told."
05:55
Now, what I'm going to do is,
05:58
I'm going to ask questions.
05:59
So, who told? I told.
06:01
I told whom? I told the carpenter.
06:04
Now we start with a different part of the sentence.
06:05
We start with the word "pay,"
06:07
and we add the ability filter to it to make it "can pay."
06:09
Then we make it "can't pay,"
06:14
and we can make it "couldn't pay"
06:16
by making it the past tense.
06:17
So who couldn't pay? I couldn't pay.
06:19
Couldn't pay whom? I couldn't pay the carpenter.
06:21
And then you join these two together
06:24
by asking this question:
06:25
What did I tell the carpenter?
06:27
I told the carpenter I could not pay him.
06:28
Now think about this. This is
06:32
—(Applause)—
06:34
this is a representation of this sentence
06:38
without language.
06:42
And there are two or three
interesting things about this.
06:44
First of all, I could have started anywhere.
06:46
I didn't have to start with the word "tell."
06:49
I could have started anywhere in the sentence,
06:52
and I could have made this entire thing.
06:53
The second thing is, if I wasn't an English speaker,
06:55
if I was speaking in some other language,
06:57
this map would actually hold true in any language.
06:59
So long as the questions are standardized,
07:03
the map is actually independent of language.
07:05
So I call this FreeSpeech,
07:09
and I was playing with this for many, many months.
07:11
I was trying out so many
different combinations of this.
07:14
And then I noticed something very
interesting about FreeSpeech.
07:17
I was trying to convert language,
07:19
convert sentences in English
into sentences in FreeSpeech,
07:22
and vice versa, and back and forth.
07:25
And I realized that this particular configuration,
07:26
this particular way of representing language,
07:29
it allowed me to actually create very concise rules
07:31
that go between FreeSpeech on one side
07:35
and English on the other.
07:38
So I could actually write this set of rules
07:39
that translates from this particular
representation into English.
07:41
And so I developed this thing.
07:45
I developed this thing called
the FreeSpeech Engine
07:47
which takes any FreeSpeech sentence as the input
07:49
and gives out perfectly grammatical English text.
07:51
And by putting these two pieces together,
07:55
the representation and the engine,
07:57
I was able to create an app, a
technology for children with autism,
07:59
that not only gives them words
08:03
but also gives them grammar.
08:05
So I tried this out with kids with autism,
08:09
and I found that there was an
incredible amount of identification.
08:11
They were able to create sentences in FreeSpeech
08:16
which were much more complicated
but much more effective
08:19
than equivalent sentences in English,
08:22
and I started thinking about
08:25
why that might be the case.
08:26
And I had an idea, and I want to
talk to you about this idea next.
08:28
In about 1997, about 15 years back,
08:33
there were a group of scientists that were trying
08:36
to understand how the brain processes language,
08:38
and they found something very interesting.
08:40
They found that when you learn a language
08:42
as a child, as a two-year-old,
08:44
you learn it with a certain part of your brain,
08:47
and when you learn a language as an adult --
08:49
for example, if I wanted to
learn Japanese right now —
08:51
a completely different part of my brain is used.
08:55
Now I don't know why that's the case,
08:57
but my guess is that that's because
08:59
when you learn a language as an adult,
09:01
you almost invariably learn it
09:04
through your native language, or
through your first language.
09:05
So what's interesting about FreeSpeech
09:09
is that when you create a sentence
09:13
or when you create language,
09:14
a child with autism creates
language with FreeSpeech,
09:16
they're not using this support language,
09:19
they're not using this bridge language.
09:21
They're directly constructing the sentence.
09:23
And so this gave me this idea.
09:26
Is it possible to use FreeSpeech
09:28
not for children with autism
09:30
but to teach language to people without disabilities?
09:33
And so I tried a number of experiments.
09:39
The first thing I did was I built a jigsaw puzzle
09:41
in which these questions and answers
09:44
are coded in the form of shapes,
09:46
in the form of colors,
09:48
and you have people putting these together
09:49
and trying to understand how this works.
09:51
And I built an app out of it, a game out of it,
09:52
in which children can play with words
09:55
and with a reinforcement,
09:57
a sound reinforcement of visual structures,
09:59
they're able to learn language.
10:02
And this, this has a lot of potential, a lot of promise,
10:04
and the government of India recently
10:06
licensed this technology from us,
10:08
and they're going to try it out
with millions of different children
10:10
trying to teach them English.
10:12
And the dream, the hope, the vision, really,
10:15
is that when they learn English this way,
10:17
they learn it with the same proficiency
10:20
as their mother tongue.
10:23
All right, let's talk about something else.
10:27
Let's talk about speech.
10:30
This is speech.
10:32
So speech is the primary mode of communication
10:34
delivered between all of us.
10:36
Now what's interesting about speech is that
10:37
speech is one-dimensional.
10:39
Why is it one-dimensional?
10:40
It's one-dimensional because it's sound.
10:42
It's also one-dimensional because
10:43
our mouths are built that way.
10:45
Our mouths are built to create
one-dimensional sound.
10:46
But if you think about the brain,
10:50
the thoughts that we have in our heads
10:52
are not one-dimensional.
10:54
I mean, we have these rich,
10:56
complicated, multi-dimensional ideas.
10:58
Now, it seems to me that language
11:01
is really the brain's invention
11:02
to convert this rich, multi-dimensional thought
11:05
on one hand
11:08
into speech on the other hand.
11:09
Now what's interesting is that
11:11
we do a lot of work in information nowadays,
11:13
and almost all of that is done
in the language domain.
11:16
Take Google, for example.
11:19
Google trawls all these
countless billions of websites,
11:21
all of which are in English,
and when you want to use Google,
11:23
you go into Google search, and you type in English,
11:26
and it matches the English with the English.
11:29
What if we could do this in FreeSpeech instead?
11:33
I have a suspicion that if we did this,
11:36
we'd find that algorithms like searching,
11:39
like retrieval, all of these things,
11:41
are much simpler and also more effective,
11:43
because they don't process
the data structure of speech.
11:46
Instead they're processing
the data structure of thought.
11:51
The data structure of thought.
11:57
That's a provocative idea.
11:59
But let's look at this in a little more detail.
12:01
So this is the FreeSpeech ecosystem.
12:04
We have the Free Speech
representation on one side,
12:06
and we have the FreeSpeech
Engine, which generates English.
12:09
Now if you think about it,
12:11
FreeSpeech, I told you, is completely
language-independent.
12:13
It doesn't have any specific information in it
12:15
which is about English.
12:17
So everything that this system knows about English
12:19
is actually encoded into the engine.
12:21
That's a pretty interesting concept in itself.
12:26
You've encoded an entire human language
12:28
into a software program.
12:32
But if you look at what's inside the engine,
12:35
it's actually not very complicated.
12:37
It's not very complicated code.
12:39
And what's more interesting is the fact that
12:41
the vast majority of the code in that engine
12:44
is not really English-specific.
12:46
And that gives this interesting idea.
12:49
It might be very easy for us to actually
12:51
create these engines in many,
many different languages,
12:53
in Hindi, in French, in German, in Swahili.
12:57
And that gives another interesting idea.
13:03
For example, supposing I was a writer,
13:06
say, for a newspaper or for a magazine.
13:08
I could create content in one language, FreeSpeech,
13:10
and the person who's consuming that content,
13:15
the person who's reading that particular information
13:18
could choose any engine,
13:21
and they could read it in their own mother tongue,
13:23
in their native language.
13:26
I mean, this is an incredibly attractive idea,
13:30
especially for India.
13:32
We have so many different languages.
13:34
There's a song about India, and there's a description
13:36
of the country as, it says,
13:38
(in Sanskrit).
13:41
That means "ever-smiling speaker
13:43
of beautiful languages."
13:46
Language is beautiful.
13:50
I think it's the most beautiful of human creations.
13:52
I think it's the loveliest thing
that our brains have invented.
13:55
It entertains, it educates, it enlightens,
13:59
but what I like the most about language
14:02
is that it empowers.
14:04
I want to leave you with this.
14:06
This is a photograph of my collaborators,
14:08
my earliest collaborators
14:10
when I started working on language
14:11
and autism and various other things.
14:13
The girl's name is Pavna,
14:14
and that's her mother, Kalpana.
14:15
And Pavna's an entrepreneur,
14:17
but her story is much more remarkable than mine,
14:19
because Pavna is about 23.
14:22
She has quadriplegic cerebral palsy,
14:24
so ever since she was born,
14:27
she could neither move nor talk.
14:28
And everything that she's accomplished so far,
14:32
finishing school, going to college,
14:34
starting a company,
14:37
collaborating with me to develop Avaz,
14:38
all of these things she's done
14:40
with nothing more than moving her eyes.
14:42
Daniel Webster said this:
14:48
He said, "If all of my possessions were taken
14:50
from me with one exception,
14:53
I would choose to keep the power of communication,
14:56
for with it, I would regain all the rest."
14:59
And that's why, of all of these incredible
applications of FreeSpeech,
15:03
the one that's closest to my heart
15:08
still remains the ability for this
15:10
to empower children with disabilities
15:12
to be able to communicate,
15:15
the power of communication,
15:17
to get back all the rest.
15:18
Thank you.
15:21
(Applause)
15:22
Thank you. (Applause)
15:23
Thank you. Thank you. Thank you. (Applause)
15:28
Thank you. Thank you. Thank you. (Applause)
15:33

sponsored links

Ajit Narayanan - Visual grammar engine inventor
Ajit Narayanan is the inventor of Avaz, an affordable, tablet-based communication device for people who are speech-impaired.

Why you should listen

Ajit Narayanan is the founder and CEO of Invention Labs, and the inventor of Avaz AAC, the first assistive device aimed at an Indian market that helps people with speech disabilities -- such as cerebral palsy, autism, intellectual disability, aphasia and learning disabilities -- to communicate. Avaz is also available as an iPad app, aimed at children with autism. In 2010, Avaz won the National Award for Empowerment of People with Disabilities from the president of India, and in 2011, Narayanan was listed in MIT Technology Review 35 under 35.
 
Narayanan is a prolific inventor with more than 20 patent applications. He is an electrical engineer with degrees from IIT Madras. His research interests are embedded systems, signal processing and understanding how the brain perceives language and communication.

sponsored links

If you need translations, you can install "Google Translate" extension into your Chrome Browser.
Furthermore, you can change playback rate by installing "Video Speed Controller" extension.

Data provided by TED.

This website is owned and operated by Tokyo English Network.
The developer's blog is here.