TED2013
Ajit Narayanan: A word game to communicate in any language
Filmed:
Readability: 3.4
1,391,245 views
While working with kids who have trouble speaking, Ajit Narayanan sketched out a way to think about language in pictures, to relate words and concepts in "maps." The idea now powers the FreeSpeech app, which can help nonverbal people communicate.
Ajit Narayanan - Visual grammar engine inventor
Ajit Narayanan is the inventor of Avaz, an affordable, tablet-based communication device for people who are speech-impaired. Full bio
Ajit Narayanan is the inventor of Avaz, an affordable, tablet-based communication device for people who are speech-impaired. Full bio
Double-click the English transcript below to play the video.
00:12
I work with children with autism.
0
721
2670
00:15
Specifically, I make technologies
1
3391
1914
00:17
to help them communicate.
2
5305
2171
00:19
Now, many of the problems that children
3
7476
1539
00:21
with autism face, they have a common source,
4
9015
3763
00:24
and that source is that they find it difficult
5
12778
2094
00:26
to understand abstraction, symbolism.
6
14872
5260
00:32
And because of this, they have
a lot of difficulty with language.
a lot of difficulty with language.
7
20132
4652
00:36
Let me tell you a little bit about why this is.
8
24784
3015
00:39
You see that this is a picture of a bowl of soup.
9
27799
3934
00:43
All of us can see it. All of us understand this.
10
31733
2485
00:46
These are two other pictures of soup,
11
34218
2312
00:48
but you can see that these are more abstract
12
36530
2067
00:50
These are not quite as concrete.
13
38597
1856
00:52
And when you get to language,
14
40453
2174
00:54
you see that it becomes a word
15
42627
1868
00:56
whose look, the way it looks and the way it sounds,
16
44495
3261
00:59
has absolutely nothing to do
with what it started with,
with what it started with,
17
47756
2912
01:02
or what it represents, which is the bowl of soup.
18
50668
2830
01:05
So it's essentially a completely abstract,
19
53498
2900
01:08
a completely arbitrary representation of something
20
56398
2576
01:10
which is in the real world,
21
58974
1163
01:12
and this is something that children with autism
22
60137
1791
01:13
have an incredible amount of difficulty with.
23
61928
3164
01:17
Now that's why most of the people
that work with children with autism --
that work with children with autism --
24
65092
2751
01:19
speech therapists, educators --
25
67843
1878
01:21
what they do is, they try to help children with autism
26
69721
2633
01:24
communicate not with words, but with pictures.
27
72354
3229
01:27
So if a child with autism wanted to say,
28
75583
1930
01:29
"I want soup," that child would pick
29
77513
2458
01:31
three different pictures, "I," "want," and "soup,"
30
79971
2260
01:34
and they would put these together,
31
82231
1609
01:35
and then the therapist or the parent would
32
83840
1867
01:37
understand that this is what the kid wants to say.
33
85707
1887
01:39
And this has been incredibly effective;
34
87594
1778
01:41
for the last 30, 40 years
35
89372
2141
01:43
people have been doing this.
36
91513
1613
01:45
In fact, a few years back,
37
93126
1349
01:46
I developed an app for the iPad
38
94475
2675
01:49
which does exactly this. It's called Avaz,
39
97150
2255
01:51
and the way it works is that kids select
40
99405
2279
01:53
different pictures.
41
101684
1321
01:55
These pictures are sequenced
together to form sentences,
together to form sentences,
42
103005
2570
01:57
and these sentences are spoken out.
43
105575
1719
01:59
So Avaz is essentially converting pictures,
44
107294
3025
02:02
it's a translator, it converts pictures into speech.
45
110319
3960
02:06
Now, this was very effective.
46
114279
1718
02:07
There are thousands of children using this,
47
115997
1384
02:09
you know, all over the world,
48
117381
1430
02:10
and I started thinking about
49
118811
2175
02:12
what it does and what it doesn't do.
50
120986
2654
02:15
And I realized something interesting:
51
123640
1684
02:17
Avaz helps children with autism learn words.
52
125324
4203
02:21
What it doesn't help them do is to learn
53
129527
2405
02:23
word patterns.
54
131932
2748
02:26
Let me explain this in a little more detail.
55
134680
2472
02:29
Take this sentence: "I want soup tonight."
56
137152
3057
02:32
Now it's not just the words
here that convey the meaning.
here that convey the meaning.
57
140209
4080
02:36
It's also the way in which these words are arranged,
58
144289
3140
02:39
the way these words are modified and arranged.
59
147429
2515
02:41
And that's why a sentence like "I want soup tonight"
60
149959
2306
02:44
is different from a sentence like
61
152265
1984
02:46
"Soup want I tonight," which
is completely meaningless.
is completely meaningless.
62
154249
3312
02:49
So there is another hidden abstraction here
63
157561
2619
02:52
which children with autism find
a lot of difficulty coping with,
a lot of difficulty coping with,
64
160180
3557
02:55
and that's the fact that you can modify words
65
163737
2840
02:58
and you can arrange them to have
66
166577
2101
03:00
different meanings, to convey different ideas.
67
168678
2895
03:03
Now, this is what we call grammar.
68
171573
3459
03:07
And grammar is incredibly powerful,
69
175032
2036
03:09
because grammar is this one component of language
70
177068
3157
03:12
which takes this finite vocabulary that all of us have
71
180225
3489
03:15
and allows us to convey an
infinite amount of information,
infinite amount of information,
72
183714
4531
03:20
an infinite amount of ideas.
73
188245
2134
03:22
It's the way in which you can put things together
74
190379
2002
03:24
in order to convey anything you want to.
75
192381
2168
03:26
And so after I developed Avaz,
76
194549
2127
03:28
I worried for a very long time
77
196676
1568
03:30
about how I could give grammar
to children with autism.
to children with autism.
78
198244
3910
03:34
The solution came to me from
a very interesting perspective.
a very interesting perspective.
79
202154
2275
03:36
I happened to chance upon a child with autism
80
204429
3449
03:39
conversing with her mom,
81
207878
2109
03:41
and this is what happened.
82
209987
2094
03:44
Completely out of the blue, very spontaneously,
83
212081
2186
03:46
the child got up and said, "Eat."
84
214267
2463
03:48
Now what was interesting was
85
216730
1770
03:50
the way in which the mom was trying to tease out
86
218500
4244
03:54
the meaning of what the child wanted to say
87
222744
2213
03:56
by talking to her in questions.
88
224957
2260
03:59
So she asked, "Eat what? Do
you want to eat ice cream?
you want to eat ice cream?
89
227217
2593
04:01
You want to eat? Somebody else wants to eat?
90
229810
2112
04:03
You want to eat cream now? You
want to eat ice cream in the evening?"
want to eat ice cream in the evening?"
91
231922
3313
04:07
And then it struck me that
92
235235
1514
04:08
what the mother had done was something incredible.
93
236749
2028
04:10
She had been able to get that child to communicate
94
238777
1994
04:12
an idea to her without grammar.
95
240771
4138
04:16
And it struck me that maybe this is what
96
244909
2696
04:19
I was looking for.
97
247605
1385
04:20
Instead of arranging words in an order, in sequence,
98
248990
4142
04:25
as a sentence, you arrange them
99
253132
2172
04:27
in this map, where they're all linked together
100
255304
3811
04:31
not by placing them one after the other
101
259115
2143
04:33
but in questions, in question-answer pairs.
102
261258
3284
04:36
And so if you do this, then what you're conveying
103
264542
2358
04:38
is not a sentence in English,
104
266900
1986
04:40
but what you're conveying is really a meaning,
105
268886
2966
04:43
the meaning of a sentence in English.
106
271852
1511
04:45
Now, meaning is really the underbelly,
in some sense, of language.
in some sense, of language.
107
273363
2932
04:48
It's what comes after thought but before language.
108
276295
3821
04:52
And the idea was that this particular representation
109
280116
2503
04:54
might convey meaning in its raw form.
110
282619
3261
04:57
So I was very excited by this, you know,
111
285880
1771
04:59
hopping around all over the place,
112
287651
1493
05:01
trying to figure out if I can convert
113
289144
1771
05:02
all possible sentences that I hear into this.
114
290915
2524
05:05
And I found that this is not enough.
115
293439
1773
05:07
Why is this not enough?
116
295212
1385
05:08
This is not enough because if you wanted to convey
117
296597
1711
05:10
something like negation,
118
298308
2250
05:12
you want to say, "I don't want soup,"
119
300558
1736
05:14
then you can't do that by asking a question.
120
302294
2220
05:16
You do that by changing the word "want."
121
304514
2285
05:18
Again, if you wanted to say,
122
306799
1637
05:20
"I wanted soup yesterday,"
123
308436
1980
05:22
you do that by converting
the word "want" into "wanted."
the word "want" into "wanted."
124
310416
2737
05:25
It's a past tense.
125
313153
1666
05:26
So this is a flourish which I added
126
314819
2103
05:28
to make the system complete.
127
316922
1576
05:30
This is a map of words joined together
128
318498
1977
05:32
as questions and answers,
129
320475
1656
05:34
and with these filters applied on top of them
130
322131
2264
05:36
in order to modify them to represent
131
324395
1817
05:38
certain nuances.
132
326212
1709
05:39
Let me show you this with a different example.
133
327921
1951
05:41
Let's take this sentence:
134
329872
1254
05:43
"I told the carpenter I could not pay him."
135
331126
1980
05:45
It's a fairly complicated sentence.
136
333106
1792
05:46
The way that this particular system works,
137
334898
1893
05:48
you can start with any part of this sentence.
138
336791
2578
05:51
I'm going to start with the word "tell."
139
339369
1698
05:53
So this is the word "tell."
140
341067
1462
05:54
Now this happened in the past,
141
342529
1600
05:56
so I'm going to make that "told."
142
344129
2223
05:58
Now, what I'm going to do is,
143
346352
1708
06:00
I'm going to ask questions.
144
348060
1756
06:01
So, who told? I told.
145
349816
2364
06:04
I told whom? I told the carpenter.
146
352180
1927
06:06
Now we start with a different part of the sentence.
147
354107
1751
06:07
We start with the word "pay,"
148
355858
1867
06:09
and we add the ability filter to it to make it "can pay."
149
357725
4577
06:14
Then we make it "can't pay,"
150
362302
2101
06:16
and we can make it "couldn't pay"
151
364403
1599
06:18
by making it the past tense.
152
366002
1663
06:19
So who couldn't pay? I couldn't pay.
153
367665
1923
06:21
Couldn't pay whom? I couldn't pay the carpenter.
154
369588
2676
06:24
And then you join these two together
155
372264
1731
06:25
by asking this question:
156
373995
1350
06:27
What did I tell the carpenter?
157
375345
1737
06:29
I told the carpenter I could not pay him.
158
377082
4049
06:33
Now think about this. This is
159
381131
1937
06:35
—(Applause)—
160
383068
3542
06:38
this is a representation of this sentence
161
386610
3672
06:42
without language.
162
390282
2435
06:44
And there are two or three
interesting things about this.
interesting things about this.
163
392717
2192
06:46
First of all, I could have started anywhere.
164
394909
3131
06:50
I didn't have to start with the word "tell."
165
398040
2243
06:52
I could have started anywhere in the sentence,
166
400283
1416
06:53
and I could have made this entire thing.
167
401699
1507
06:55
The second thing is, if I wasn't an English speaker,
168
403206
2776
06:57
if I was speaking in some other language,
169
405982
2175
07:00
this map would actually hold true in any language.
170
408157
3156
07:03
So long as the questions are standardized,
171
411313
1990
07:05
the map is actually independent of language.
172
413303
4287
07:09
So I call this FreeSpeech,
173
417590
2115
07:11
and I was playing with this for many, many months.
174
419705
2935
07:14
I was trying out so many
different combinations of this.
different combinations of this.
175
422640
2726
07:17
And then I noticed something very
interesting about FreeSpeech.
interesting about FreeSpeech.
176
425366
2289
07:19
I was trying to convert language,
177
427655
3243
07:22
convert sentences in English
into sentences in FreeSpeech,
into sentences in FreeSpeech,
178
430898
2384
07:25
and vice versa, and back and forth.
179
433282
1752
07:27
And I realized that this particular configuration,
180
435034
2255
07:29
this particular way of representing language,
181
437289
2026
07:31
it allowed me to actually create very concise rules
182
439315
4395
07:35
that go between FreeSpeech on one side
183
443710
2734
07:38
and English on the other.
184
446444
1488
07:39
So I could actually write this set of rules
185
447932
2180
07:42
that translates from this particular
representation into English.
representation into English.
186
450112
3395
07:45
And so I developed this thing.
187
453507
1831
07:47
I developed this thing called
the FreeSpeech Engine
the FreeSpeech Engine
188
455338
2232
07:49
which takes any FreeSpeech sentence as the input
189
457570
2561
07:52
and gives out perfectly grammatical English text.
190
460131
3930
07:56
And by putting these two pieces together,
191
464061
1605
07:57
the representation and the engine,
192
465666
1881
07:59
I was able to create an app, a
technology for children with autism,
technology for children with autism,
193
467547
3796
08:03
that not only gives them words
194
471343
2499
08:05
but also gives them grammar.
195
473842
3941
08:09
So I tried this out with kids with autism,
196
477783
2360
08:12
and I found that there was an
incredible amount of identification.
incredible amount of identification.
197
480143
5013
08:17
They were able to create sentences in FreeSpeech
198
485156
2720
08:19
which were much more complicated
but much more effective
but much more effective
199
487876
2558
08:22
than equivalent sentences in English,
200
490434
2899
08:25
and I started thinking about
201
493333
1682
08:27
why that might be the case.
202
495015
1969
08:28
And I had an idea, and I want to
talk to you about this idea next.
talk to you about this idea next.
203
496984
4287
08:33
In about 1997, about 15 years back,
204
501271
3142
08:36
there were a group of scientists that were trying
205
504413
2011
08:38
to understand how the brain processes language,
206
506424
2389
08:40
and they found something very interesting.
207
508813
1779
08:42
They found that when you learn a language
208
510592
1872
08:44
as a child, as a two-year-old,
209
512464
2912
08:47
you learn it with a certain part of your brain,
210
515376
2366
08:49
and when you learn a language as an adult --
211
517742
1600
08:51
for example, if I wanted to
learn Japanese right now —
learn Japanese right now —
212
519342
3911
08:55
a completely different part of my brain is used.
213
523253
2707
08:57
Now I don't know why that's the case,
214
525960
1831
08:59
but my guess is that that's because
215
527791
1991
09:01
when you learn a language as an adult,
216
529782
2437
09:04
you almost invariably learn it
217
532219
1616
09:05
through your native language, or
through your first language.
through your first language.
218
533835
4266
09:10
So what's interesting about FreeSpeech
219
538101
3252
09:13
is that when you create a sentence
220
541353
1802
09:15
or when you create language,
221
543155
1695
09:16
a child with autism creates
language with FreeSpeech,
language with FreeSpeech,
222
544850
3070
09:19
they're not using this support language,
223
547920
1833
09:21
they're not using this bridge language.
224
549753
2211
09:23
They're directly constructing the sentence.
225
551964
2657
09:26
And so this gave me this idea.
226
554621
2193
09:28
Is it possible to use FreeSpeech
227
556814
2024
09:30
not for children with autism
228
558838
2510
09:33
but to teach language to people without disabilities?
229
561348
6262
09:39
And so I tried a number of experiments.
230
567610
1978
09:41
The first thing I did was I built a jigsaw puzzle
231
569588
2948
09:44
in which these questions and answers
232
572536
1970
09:46
are coded in the form of shapes,
233
574506
1835
09:48
in the form of colors,
234
576341
1138
09:49
and you have people putting these together
235
577479
1849
09:51
and trying to understand how this works.
236
579328
1773
09:53
And I built an app out of it, a game out of it,
237
581101
2376
09:55
in which children can play with words
238
583477
2661
09:58
and with a reinforcement,
239
586138
1704
09:59
a sound reinforcement of visual structures,
240
587842
2585
10:02
they're able to learn language.
241
590427
2013
10:04
And this, this has a lot of potential, a lot of promise,
242
592440
2736
10:07
and the government of India recently
243
595176
1975
10:09
licensed this technology from us,
244
597151
1404
10:10
and they're going to try it out
with millions of different children
with millions of different children
245
598555
2074
10:12
trying to teach them English.
246
600629
2605
10:15
And the dream, the hope, the vision, really,
247
603234
2614
10:17
is that when they learn English this way,
248
605848
3082
10:20
they learn it with the same proficiency
249
608930
2643
10:23
as their mother tongue.
250
611573
3718
10:27
All right, let's talk about something else.
251
615291
3816
10:31
Let's talk about speech.
252
619107
1997
10:33
This is speech.
253
621104
1271
10:34
So speech is the primary mode of communication
254
622375
1962
10:36
delivered between all of us.
255
624337
1613
10:37
Now what's interesting about speech is that
256
625950
1855
10:39
speech is one-dimensional.
257
627805
1245
10:41
Why is it one-dimensional?
258
629050
1359
10:42
It's one-dimensional because it's sound.
259
630409
1568
10:43
It's also one-dimensional because
260
631977
1539
10:45
our mouths are built that way.
261
633516
1205
10:46
Our mouths are built to create
one-dimensional sound.
one-dimensional sound.
262
634721
3512
10:50
But if you think about the brain,
263
638233
2866
10:53
the thoughts that we have in our heads
264
641099
1764
10:54
are not one-dimensional.
265
642863
2102
10:56
I mean, we have these rich,
266
644965
1459
10:58
complicated, multi-dimensional ideas.
267
646424
3028
11:01
Now, it seems to me that language
268
649452
1690
11:03
is really the brain's invention
269
651142
2332
11:05
to convert this rich, multi-dimensional thought
270
653474
3096
11:08
on one hand
271
656570
1587
11:10
into speech on the other hand.
272
658157
1923
11:12
Now what's interesting is that
273
660080
1762
11:13
we do a lot of work in information nowadays,
274
661842
2568
11:16
and almost all of that is done
in the language domain.
in the language domain.
275
664410
3079
11:19
Take Google, for example.
276
667489
1939
11:21
Google trawls all these
countless billions of websites,
countless billions of websites,
277
669428
2677
11:24
all of which are in English,
and when you want to use Google,
and when you want to use Google,
278
672105
2725
11:26
you go into Google search, and you type in English,
279
674830
2450
11:29
and it matches the English with the English.
280
677280
4163
11:33
What if we could do this in FreeSpeech instead?
281
681443
3583
11:37
I have a suspicion that if we did this,
282
685026
2301
11:39
we'd find that algorithms like searching,
283
687327
2068
11:41
like retrieval, all of these things,
284
689395
2325
11:43
are much simpler and also more effective,
285
691720
3075
11:46
because they don't process
the data structure of speech.
the data structure of speech.
286
694795
4417
11:51
Instead they're processing
the data structure of thought.
the data structure of thought.
287
699212
5976
11:57
The data structure of thought.
288
705188
2808
11:59
That's a provocative idea.
289
707996
2076
12:02
But let's look at this in a little more detail.
290
710072
2142
12:04
So this is the FreeSpeech ecosystem.
291
712214
2366
12:06
We have the Free Speech
representation on one side,
representation on one side,
292
714580
2884
12:09
and we have the FreeSpeech
Engine, which generates English.
Engine, which generates English.
293
717464
2228
12:11
Now if you think about it,
294
719694
1725
12:13
FreeSpeech, I told you, is completely
language-independent.
language-independent.
295
721419
2544
12:15
It doesn't have any specific information in it
296
723963
2087
12:18
which is about English.
297
726050
1228
12:19
So everything that this system knows about English
298
727278
2800
12:22
is actually encoded into the engine.
299
730078
4620
12:26
That's a pretty interesting concept in itself.
300
734698
2237
12:28
You've encoded an entire human language
301
736935
3604
12:32
into a software program.
302
740539
2645
12:35
But if you look at what's inside the engine,
303
743184
2531
12:37
it's actually not very complicated.
304
745715
2358
12:40
It's not very complicated code.
305
748073
2105
12:42
And what's more interesting is the fact that
306
750178
2672
12:44
the vast majority of the code in that engine
307
752850
2203
12:47
is not really English-specific.
308
755053
2412
12:49
And that gives this interesting idea.
309
757465
1895
12:51
It might be very easy for us to actually
310
759360
2038
12:53
create these engines in many,
many different languages,
many different languages,
311
761398
3826
12:57
in Hindi, in French, in German, in Swahili.
312
765224
6354
13:03
And that gives another interesting idea.
313
771578
2799
13:06
For example, supposing I was a writer,
314
774377
2654
13:09
say, for a newspaper or for a magazine.
315
777031
2122
13:11
I could create content in one language, FreeSpeech,
316
779153
5011
13:16
and the person who's consuming that content,
317
784164
2056
13:18
the person who's reading that particular information
318
786220
3061
13:21
could choose any engine,
319
789281
2495
13:23
and they could read it in their own mother tongue,
320
791776
2736
13:26
in their native language.
321
794512
3939
13:30
I mean, this is an incredibly attractive idea,
322
798451
2722
13:33
especially for India.
323
801173
1999
13:35
We have so many different languages.
324
803172
1690
13:36
There's a song about India, and there's a description
325
804862
2142
13:39
of the country as, it says,
326
807004
2344
13:41
(in Sanskrit).
327
809348
2360
13:43
That means "ever-smiling speaker
328
811708
2773
13:46
of beautiful languages."
329
814481
4519
13:51
Language is beautiful.
330
819000
1964
13:52
I think it's the most beautiful of human creations.
331
820964
2454
13:55
I think it's the loveliest thing
that our brains have invented.
that our brains have invented.
332
823418
3978
13:59
It entertains, it educates, it enlightens,
333
827396
3584
14:02
but what I like the most about language
334
830980
2044
14:05
is that it empowers.
335
833024
1500
14:06
I want to leave you with this.
336
834524
1838
14:08
This is a photograph of my collaborators,
337
836362
2385
14:10
my earliest collaborators
338
838747
997
14:11
when I started working on language
339
839744
1462
14:13
and autism and various other things.
340
841206
1502
14:14
The girl's name is Pavna,
341
842708
1417
14:16
and that's her mother, Kalpana.
342
844125
1902
14:18
And Pavna's an entrepreneur,
343
846027
2138
14:20
but her story is much more remarkable than mine,
344
848165
2371
14:22
because Pavna is about 23.
345
850536
2400
14:24
She has quadriplegic cerebral palsy,
346
852936
2552
14:27
so ever since she was born,
347
855488
1640
14:29
she could neither move nor talk.
348
857128
3600
14:32
And everything that she's accomplished so far,
349
860728
2403
14:35
finishing school, going to college,
350
863131
2227
14:37
starting a company,
351
865358
1416
14:38
collaborating with me to develop Avaz,
352
866774
2140
14:40
all of these things she's done
353
868914
1892
14:42
with nothing more than moving her eyes.
354
870806
5523
14:48
Daniel Webster said this:
355
876329
2689
14:51
He said, "If all of my possessions were taken
356
879018
2940
14:53
from me with one exception,
357
881958
2988
14:56
I would choose to keep the power of communication,
358
884946
2981
14:59
for with it, I would regain all the rest."
359
887927
3903
15:03
And that's why, of all of these incredible
applications of FreeSpeech,
applications of FreeSpeech,
360
891830
5116
15:08
the one that's closest to my heart
361
896946
2080
15:11
still remains the ability for this
362
899026
2068
15:13
to empower children with disabilities
363
901094
2380
15:15
to be able to communicate,
364
903474
1773
15:17
the power of communication,
365
905247
1789
15:19
to get back all the rest.
366
907036
2240
15:21
Thank you.
367
909276
1397
15:22
(Applause)
368
910673
1332
15:24
Thank you. (Applause)
369
912005
4199
15:28
Thank you. Thank you. Thank you. (Applause)
370
916204
5323
15:33
Thank you. Thank you. Thank you. (Applause)
371
921527
4000
ABOUT THE SPEAKER
Ajit Narayanan - Visual grammar engine inventorAjit Narayanan is the inventor of Avaz, an affordable, tablet-based communication device for people who are speech-impaired.
Why you should listen
Ajit Narayanan is the founder and CEO of Invention Labs, and the inventor of Avaz AAC, the first assistive device aimed at an Indian market that helps people with speech disabilities -- such as cerebral palsy, autism, intellectual disability, aphasia and learning disabilities -- to communicate. Avaz is also available as an iPad app, aimed at children with autism. In 2010, Avaz won the National Award for Empowerment of People with Disabilities from the president of India, and in 2011, Narayanan was listed in MIT Technology Review 35 under 35.
Narayanan is a prolific inventor with more than 20 patent applications. He is an electrical engineer with degrees from IIT Madras. His research interests are embedded systems, signal processing and understanding how the brain perceives language and communication.
Ajit Narayanan | Speaker | TED.com