TEDWomen 2015
Rana el Kaliouby: This app knows how you feel -- from the look on your face
Filmed:
Readability: 4.3
1,613,290 views
Our emotions influence every aspect of our lives -- how we learn, how we communicate, how we make decisions. Yet they're absent from our digital lives; the devices and apps we interact with have no way of knowing how we feel. Scientist Rana el Kaliouby aims to change that. She demos a powerful new technology that reads your facial expressions and matches them to corresponding emotions. This "emotion engine" has big implications, she says, and could change not just how we interact with machines -- but with each other.
Rana el Kaliouby - Computer scientist
What if a computer could recognize your facial expression, and react to how you feel? Rana el Kaliouby sees big possibilities in making technology emotionally aware. Full bio
What if a computer could recognize your facial expression, and react to how you feel? Rana el Kaliouby sees big possibilities in making technology emotionally aware. Full bio
Double-click the English transcript below to play the video.
00:12
Our emotions influence
every aspect of our lives,
every aspect of our lives,
0
556
4017
00:16
from our health and how we learn,
to how we do business and make decisions,
to how we do business and make decisions,
1
4573
3576
00:20
big ones and small.
2
8149
1773
00:22
Our emotions also influence
how we connect with one another.
how we connect with one another.
3
10672
3490
00:27
We've evolved to live
in a world like this,
in a world like this,
4
15132
3976
00:31
but instead, we're living
more and more of our lives like this --
more and more of our lives like this --
5
19108
4319
00:35
this is the text message
from my daughter last night --
from my daughter last night --
6
23427
3134
00:38
in a world that's devoid of emotion.
7
26561
2740
00:41
So I'm on a mission to change that.
8
29301
1951
00:43
I want to bring emotions
back into our digital experiences.
back into our digital experiences.
9
31252
4091
00:48
I started on this path 15 years ago.
10
36223
3077
00:51
I was a computer scientist in Egypt,
11
39300
2066
00:53
and I had just gotten accepted to
a Ph.D. program at Cambridge University.
a Ph.D. program at Cambridge University.
12
41366
4505
00:57
So I did something quite unusual
13
45871
2113
00:59
for a young newlywed Muslim Egyptian wife:
14
47984
4225
01:05
With the support of my husband,
who had to stay in Egypt,
who had to stay in Egypt,
15
53599
2999
01:08
I packed my bags and I moved to England.
16
56598
3018
01:11
At Cambridge, thousands of miles
away from home,
away from home,
17
59616
3228
01:14
I realized I was spending
more hours with my laptop
more hours with my laptop
18
62844
3413
01:18
than I did with any other human.
19
66257
2229
01:20
Yet despite this intimacy, my laptop
had absolutely no idea how I was feeling.
had absolutely no idea how I was feeling.
20
68486
4853
01:25
It had no idea if I was happy,
21
73339
3211
01:28
having a bad day, or stressed, confused,
22
76550
2988
01:31
and so that got frustrating.
23
79538
2922
01:35
Even worse, as I communicated
online with my family back home,
online with my family back home,
24
83600
5231
01:41
I felt that all my emotions
disappeared in cyberspace.
disappeared in cyberspace.
25
89421
3282
01:44
I was homesick, I was lonely,
and on some days I was actually crying,
and on some days I was actually crying,
26
92703
5155
01:49
but all I had to communicate
these emotions was this.
these emotions was this.
27
97858
4928
01:54
(Laughter)
28
102786
2020
01:56
Today's technology
has lots of I.Q., but no E.Q.;
has lots of I.Q., but no E.Q.;
29
104806
4974
02:01
lots of cognitive intelligence,
but no emotional intelligence.
but no emotional intelligence.
30
109780
3176
02:04
So that got me thinking,
31
112956
2197
02:07
what if our technology
could sense our emotions?
could sense our emotions?
32
115153
3624
02:10
What if our devices could sense
how we felt and reacted accordingly,
how we felt and reacted accordingly,
33
118777
4076
02:14
just the way an emotionally
intelligent friend would?
intelligent friend would?
34
122853
3013
02:18
Those questions led me and my team
35
126666
3564
02:22
to create technologies that can read
and respond to our emotions,
and respond to our emotions,
36
130230
4377
02:26
and our starting point was the human face.
37
134607
3090
02:30
So our human face happens to be
one of the most powerful channels
one of the most powerful channels
38
138577
3173
02:33
that we all use to communicate
social and emotional states,
social and emotional states,
39
141750
4016
02:37
everything from enjoyment, surprise,
40
145766
3010
02:40
empathy and curiosity.
41
148776
4203
02:44
In emotion science, we call each
facial muscle movement an action unit.
facial muscle movement an action unit.
42
152979
4928
02:49
So for example, action unit 12,
43
157907
2925
02:52
it's not a Hollywood blockbuster,
44
160832
2038
02:54
it is actually a lip corner pull,
which is the main component of a smile.
which is the main component of a smile.
45
162870
3442
02:58
Try it everybody. Let's get
some smiles going on.
some smiles going on.
46
166312
2988
03:01
Another example is action unit 4.
It's the brow furrow.
It's the brow furrow.
47
169300
2654
03:03
It's when you draw your eyebrows together
48
171954
2238
03:06
and you create all
these textures and wrinkles.
these textures and wrinkles.
49
174192
2267
03:08
We don't like them, but it's
a strong indicator of a negative emotion.
a strong indicator of a negative emotion.
50
176459
4295
03:12
So we have about 45 of these action units,
51
180754
2206
03:14
and they combine to express
hundreds of emotions.
hundreds of emotions.
52
182960
3390
03:18
Teaching a computer to read
these facial emotions is hard,
these facial emotions is hard,
53
186350
3901
03:22
because these action units,
they can be fast, they're subtle,
they can be fast, they're subtle,
54
190251
2972
03:25
and they combine in many different ways.
55
193223
2554
03:27
So take, for example,
the smile and the smirk.
the smile and the smirk.
56
195777
3738
03:31
They look somewhat similar,
but they mean very different things.
but they mean very different things.
57
199515
3753
03:35
(Laughter)
58
203268
1718
03:36
So the smile is positive,
59
204986
3004
03:39
a smirk is often negative.
60
207990
1270
03:41
Sometimes a smirk
can make you become famous.
can make you become famous.
61
209260
3876
03:45
But seriously, it's important
for a computer to be able
for a computer to be able
62
213136
2824
03:47
to tell the difference
between the two expressions.
between the two expressions.
63
215960
2855
03:50
So how do we do that?
64
218815
1812
03:52
We give our algorithms
65
220627
1787
03:54
tens of thousands of examples
of people we know to be smiling,
of people we know to be smiling,
66
222414
4110
03:58
from different ethnicities, ages, genders,
67
226524
3065
04:01
and we do the same for smirks.
68
229589
2811
04:04
And then, using deep learning,
69
232400
1554
04:05
the algorithm looks for all these
textures and wrinkles
textures and wrinkles
70
233954
2856
04:08
and shape changes on our face,
71
236810
2580
04:11
and basically learns that all smiles
have common characteristics,
have common characteristics,
72
239390
3202
04:14
all smirks have subtly
different characteristics.
different characteristics.
73
242592
3181
04:17
And the next time it sees a new face,
74
245773
2368
04:20
it essentially learns that
75
248141
2299
04:22
this face has the same
characteristics of a smile,
characteristics of a smile,
76
250440
3033
04:25
and it says, "Aha, I recognize this.
This is a smile expression."
This is a smile expression."
77
253473
4278
04:30
So the best way to demonstrate
how this technology works
how this technology works
78
258381
2800
04:33
is to try a live demo,
79
261181
2136
04:35
so I need a volunteer,
preferably somebody with a face.
preferably somebody with a face.
80
263317
3913
04:39
(Laughter)
81
267230
2334
04:41
Cloe's going to be our volunteer today.
82
269564
2771
04:45
So over the past five years, we've moved
from being a research project at MIT
from being a research project at MIT
83
273325
4458
04:49
to a company,
84
277783
1156
04:50
where my team has worked really hard
to make this technology work,
to make this technology work,
85
278939
3192
04:54
as we like to say, in the wild.
86
282131
2409
04:56
And we've also shrunk it so that
the core emotion engine
the core emotion engine
87
284540
2670
04:59
works on any mobile device
with a camera, like this iPad.
with a camera, like this iPad.
88
287210
3320
05:02
So let's give this a try.
89
290530
2786
05:06
As you can see, the algorithm
has essentially found Cloe's face,
has essentially found Cloe's face,
90
294756
3924
05:10
so it's this white bounding box,
91
298680
1692
05:12
and it's tracking the main
feature points on her face,
feature points on her face,
92
300372
2571
05:14
so her eyebrows, her eyes,
her mouth and her nose.
her mouth and her nose.
93
302943
2856
05:17
The question is,
can it recognize her expression?
can it recognize her expression?
94
305799
2987
05:20
So we're going to test the machine.
95
308786
1671
05:22
So first of all, give me your poker face.
Yep, awesome. (Laughter)
Yep, awesome. (Laughter)
96
310457
4186
05:26
And then as she smiles,
this is a genuine smile, it's great.
this is a genuine smile, it's great.
97
314643
2813
05:29
So you can see the green bar
go up as she smiles.
go up as she smiles.
98
317456
2300
05:31
Now that was a big smile.
99
319756
1222
05:32
Can you try a subtle smile
to see if the computer can recognize?
to see if the computer can recognize?
100
320978
3043
05:36
It does recognize subtle smiles as well.
101
324021
2331
05:38
We've worked really hard
to make that happen.
to make that happen.
102
326352
2125
05:40
And then eyebrow raised,
indicator of surprise.
indicator of surprise.
103
328477
2962
05:43
Brow furrow, which is
an indicator of confusion.
an indicator of confusion.
104
331439
4249
05:47
Frown. Yes, perfect.
105
335688
4007
05:51
So these are all the different
action units. There's many more of them.
action units. There's many more of them.
106
339695
3493
05:55
This is just a slimmed-down demo.
107
343188
2032
05:57
But we call each reading
an emotion data point,
an emotion data point,
108
345220
3148
06:00
and then they can fire together
to portray different emotions.
to portray different emotions.
109
348368
2969
06:03
So on the right side of the demo --
look like you're happy.
look like you're happy.
110
351337
4653
06:07
So that's joy. Joy fires up.
111
355990
1454
06:09
And then give me a disgust face.
112
357444
1927
06:11
Try to remember what it was like
when Zayn left One Direction.
when Zayn left One Direction.
113
359371
4272
06:15
(Laughter)
114
363643
1510
06:17
Yeah, wrinkle your nose. Awesome.
115
365153
4342
06:21
And the valence is actually quite
negative, so you must have been a big fan.
negative, so you must have been a big fan.
116
369495
3731
06:25
So valence is how positive
or negative an experience is,
or negative an experience is,
117
373226
2700
06:27
and engagement is how
expressive she is as well.
expressive she is as well.
118
375926
2786
06:30
So imagine if Cloe had access
to this real-time emotion stream,
to this real-time emotion stream,
119
378712
3414
06:34
and she could share it
with anybody she wanted to.
with anybody she wanted to.
120
382126
2809
06:36
Thank you.
121
384935
2923
06:39
(Applause)
122
387858
4621
06:45
So, so far, we have amassed
12 billion of these emotion data points.
12 billion of these emotion data points.
123
393749
5270
06:51
It's the largest emotion
database in the world.
database in the world.
124
399019
2611
06:53
We've collected it
from 2.9 million face videos,
from 2.9 million face videos,
125
401630
2963
06:56
people who have agreed
to share their emotions with us,
to share their emotions with us,
126
404593
2600
06:59
and from 75 countries around the world.
127
407193
3205
07:02
It's growing every day.
128
410398
1715
07:04
It blows my mind away
129
412603
2067
07:06
that we can now quantify something
as personal as our emotions,
as personal as our emotions,
130
414670
3195
07:09
and we can do it at this scale.
131
417865
2235
07:12
So what have we learned to date?
132
420100
2177
07:15
Gender.
133
423057
2331
07:17
Our data confirms something
that you might suspect.
that you might suspect.
134
425388
3646
07:21
Women are more expressive than men.
135
429034
1857
07:22
Not only do they smile more,
their smiles last longer,
their smiles last longer,
136
430891
2683
07:25
and we can now really quantify
what it is that men and women
what it is that men and women
137
433574
2904
07:28
respond to differently.
138
436478
2136
07:30
Let's do culture: So in the United States,
139
438614
2290
07:32
women are 40 percent
more expressive than men,
more expressive than men,
140
440904
3204
07:36
but curiously, we don't see any difference
in the U.K. between men and women.
in the U.K. between men and women.
141
444108
3645
07:39
(Laughter)
142
447753
2506
07:43
Age: People who are 50 years and older
143
451296
4027
07:47
are 25 percent more emotive
than younger people.
than younger people.
144
455323
3436
07:51
Women in their 20s smile a lot more
than men the same age,
than men the same age,
145
459899
3852
07:55
perhaps a necessity for dating.
146
463751
3839
07:59
But perhaps what surprised us
the most about this data
the most about this data
147
467590
2617
08:02
is that we happen
to be expressive all the time,
to be expressive all the time,
148
470207
3203
08:05
even when we are sitting
in front of our devices alone,
in front of our devices alone,
149
473410
2833
08:08
and it's not just when we're watching
cat videos on Facebook.
cat videos on Facebook.
150
476243
3274
08:12
We are expressive when we're emailing,
texting, shopping online,
texting, shopping online,
151
480217
3010
08:15
or even doing our taxes.
152
483227
2300
08:17
Where is this data used today?
153
485527
2392
08:19
In understanding how we engage with media,
154
487919
2763
08:22
so understanding virality
and voting behavior;
and voting behavior;
155
490682
2484
08:25
and also empowering
or emotion-enabling technology,
or emotion-enabling technology,
156
493166
2740
08:27
and I want to share some examples
that are especially close to my heart.
that are especially close to my heart.
157
495906
4621
08:33
Emotion-enabled wearable glasses
can help individuals
can help individuals
158
501197
3068
08:36
who are visually impaired
read the faces of others,
read the faces of others,
159
504265
3228
08:39
and it can help individuals
on the autism spectrum interpret emotion,
on the autism spectrum interpret emotion,
160
507493
4187
08:43
something that they really struggle with.
161
511680
2778
08:47
In education, imagine
if your learning apps
if your learning apps
162
515918
2859
08:50
sense that you're confused and slow down,
163
518777
2810
08:53
or that you're bored, so it's sped up,
164
521587
1857
08:55
just like a great teacher
would in a classroom.
would in a classroom.
165
523444
2969
08:59
What if your wristwatch tracked your mood,
166
527043
2601
09:01
or your car sensed that you're tired,
167
529644
2693
09:04
or perhaps your fridge
knows that you're stressed,
knows that you're stressed,
168
532337
2548
09:06
so it auto-locks to prevent you
from binge eating. (Laughter)
from binge eating. (Laughter)
169
534885
6066
09:12
I would like that, yeah.
170
540951
2717
09:15
What if, when I was in Cambridge,
171
543668
1927
09:17
I had access to my real-time
emotion stream,
emotion stream,
172
545595
2313
09:19
and I could share that with my family
back home in a very natural way,
back home in a very natural way,
173
547908
3529
09:23
just like I would've if we were all
in the same room together?
in the same room together?
174
551437
3971
09:27
I think five years down the line,
175
555408
3142
09:30
all our devices are going
to have an emotion chip,
to have an emotion chip,
176
558550
2337
09:32
and we won't remember what it was like
when we couldn't just frown at our device
when we couldn't just frown at our device
177
560887
4064
09:36
and our device would say, "Hmm,
you didn't like that, did you?"
you didn't like that, did you?"
178
564951
4249
09:41
Our biggest challenge is that there are
so many applications of this technology,
so many applications of this technology,
179
569200
3761
09:44
my team and I realize that we can't
build them all ourselves,
build them all ourselves,
180
572961
2903
09:47
so we've made this technology available
so that other developers
so that other developers
181
575864
3496
09:51
can get building and get creative.
182
579360
2114
09:53
We recognize that
there are potential risks
there are potential risks
183
581474
4086
09:57
and potential for abuse,
184
585560
2067
09:59
but personally, having spent
many years doing this,
many years doing this,
185
587627
2949
10:02
I believe that the benefits to humanity
186
590576
2972
10:05
from having emotionally
intelligent technology
intelligent technology
187
593548
2275
10:07
far outweigh the potential for misuse.
188
595823
3576
10:11
And I invite you all to be
part of the conversation.
part of the conversation.
189
599399
2531
10:13
The more people who know
about this technology,
about this technology,
190
601930
2554
10:16
the more we can all have a voice
in how it's being used.
in how it's being used.
191
604484
3177
10:21
So as more and more
of our lives become digital,
of our lives become digital,
192
609081
4574
10:25
we are fighting a losing battle
trying to curb our usage of devices
trying to curb our usage of devices
193
613655
3498
10:29
in order to reclaim our emotions.
194
617153
2229
10:32
So what I'm trying to do instead
is to bring emotions into our technology
is to bring emotions into our technology
195
620622
3914
10:36
and make our technologies more responsive.
196
624536
2229
10:38
So I want those devices
that have separated us
that have separated us
197
626765
2670
10:41
to bring us back together.
198
629435
2462
10:43
And by humanizing technology,
we have this golden opportunity
we have this golden opportunity
199
631897
4588
10:48
to reimagine how we
connect with machines,
connect with machines,
200
636485
3297
10:51
and therefore, how we, as human beings,
201
639782
4481
10:56
connect with one another.
202
644263
1904
10:58
Thank you.
203
646167
2160
11:00
(Applause)
204
648327
3313
ABOUT THE SPEAKER
Rana el Kaliouby - Computer scientistWhat if a computer could recognize your facial expression, and react to how you feel? Rana el Kaliouby sees big possibilities in making technology emotionally aware.
Why you should listen
Rana el Kaliouby, chief science officer and co-founder of Affectiva, an MIT Media Lab spin-off, is on a mission to bring emotion intelligence to our digital experiences. She leads the company's emotion analytics team, which is responsible for developing emotion-sensing algorithms and mining the world's largest emotion data database. So far, they've collected 12 billion emotion data points from 2.9 million face videos from volunteers in 75 countries. The company’s platform is used by many Fortune Global 100 companies to measure consumer engagement, and is pioneering emotion-enabled digital apps for enterprise, entertainment, video communication and online education.
Entrepreneur magazine called el Kaliouby one of “The 7 Most Powerful Women To Watch in 2014,” and the MIT Technology Review included her in their list of the “Top 35 Innovators Under 35.”
Rana el Kaliouby | Speaker | TED.com