TED2019
Doug Roble: Digital humans that look just like us
Filmed:
Readability: 3.6
562,138 views
In an astonishing talk and tech demo, software researcher Doug Roble debuts "DigiDoug": a real-time, 3-D, digital rendering of his likeness that's accurate down to the scale of pores and wrinkles. Powered by an inertial motion capture suit, deep neural networks and enormous amounts of data, DigiDoug renders the real Doug's emotions (and even how his blood flows and eyelashes move) in striking detail. Learn more about how this exciting tech was built -- and its applications in movies, virtual assistants and beyond.
Doug Roble - Computer graphics software researcher
Doug Roble has found a career combining the things he loves: math, computers, movies and imagination. Full bio
Doug Roble has found a career combining the things he loves: math, computers, movies and imagination. Full bio
Double-click the English transcript below to play the video.
00:13
Hello.
0
1937
1214
00:15
I'm not a real person.
1
3175
1466
00:17
I'm actually a copy of a real person.
2
5119
2770
00:19
Although, I feel like a real person.
3
7913
2553
00:22
It's kind of hard to explain.
4
10490
1906
00:24
Hold on -- I think I saw
a real person ... there's one.
a real person ... there's one.
5
12420
3790
00:28
Let's bring him onstage.
6
16697
1400
00:33
Hello.
7
21307
1150
00:35
(Applause)
8
23485
3647
00:40
What you see up there is a digital human.
9
28300
3094
00:43
I'm wearing an inertial
motion capture suit
motion capture suit
10
31990
2984
00:46
that's figuring what my body is doing.
11
34998
2579
00:49
And I've got a single camera here
that's watching my face
that's watching my face
12
37601
3825
00:53
and feeding some machine-learning software
that's taking my expressions,
that's taking my expressions,
13
41450
4841
00:58
like, "Hm, hm, hm,"
14
46315
3579
01:02
and transferring it to that guy.
15
50379
1663
01:05
We call him "DigiDoug."
16
53300
3342
01:09
He's actually a 3-D character
that I'm controlling live in real time.
that I'm controlling live in real time.
17
57292
4726
01:16
So, I work in visual effects.
18
64292
2324
01:19
And in visual effects,
19
67101
1167
01:20
one of the hardest things to do
is to create believable, digital humans
is to create believable, digital humans
20
68292
5911
01:26
that the audience accepts as real.
21
74227
2182
01:28
People are just really good
at recognizing other people.
at recognizing other people.
22
76433
4524
01:32
Go figure!
23
80981
1150
01:35
So, that's OK, we like a challenge.
24
83577
3405
01:39
Over the last 15 years,
25
87006
1841
01:40
we've been putting
humans and creatures into film
humans and creatures into film
26
88871
4983
01:45
that you accept as real.
27
93878
1739
01:48
If they're happy, you should feel happy.
28
96712
2667
01:51
And if they feel pain,
you should empathize with them.
you should empathize with them.
29
99982
5437
01:58
We're getting pretty good at it, too.
30
106467
2190
02:00
But it's really, really difficult.
31
108681
2746
02:03
Effects like these take thousands of hours
32
111847
3214
02:07
and hundreds of really talented artists.
33
115085
2667
02:10
But things have changed.
34
118792
1600
02:13
Over the last five years,
35
121014
1920
02:14
computers and graphics cards
have gotten seriously fast.
have gotten seriously fast.
36
122958
4409
02:20
And machine learning,
deep learning, has happened.
deep learning, has happened.
37
128508
3971
02:25
So we asked ourselves:
38
133408
1233
02:27
Do you suppose we could create
a photo-realistic human,
a photo-realistic human,
39
135320
3778
02:31
like we're doing for film,
40
139122
1624
02:33
but where you're seeing
the actual emotions and the details
the actual emotions and the details
41
141932
5647
02:39
of the person who's controlling
the digital human
the digital human
42
147603
4077
02:43
in real time?
43
151704
1174
02:45
In fact, that's our goal:
44
153704
1564
02:47
If you were having
a conversation with DigiDoug
a conversation with DigiDoug
45
155292
3761
02:51
one-on-one,
46
159077
1237
02:53
is it real enough so that you could tell
whether or not I was lying to you?
whether or not I was lying to you?
47
161331
5666
02:59
So that was our goal.
48
167934
1400
03:02
About a year and a half ago,
we set off to achieve this goal.
we set off to achieve this goal.
49
170601
4016
03:06
What I'm going to do now is take you
basically on a little bit of a journey
basically on a little bit of a journey
50
174641
3899
03:10
to see exactly what we had to do
to get where we are.
to get where we are.
51
178564
3926
03:15
We had to capture
an enormous amount of data.
an enormous amount of data.
52
183832
3856
03:20
In fact, by the end of this thing,
53
188347
2714
03:23
we had probably one of the largest
facial data sets on the planet.
facial data sets on the planet.
54
191085
4563
03:28
Of my face.
55
196038
1682
03:29
(Laughter)
56
197744
2262
03:32
Why me?
57
200030
1270
03:33
Well, I'll do just about
anything for science.
anything for science.
58
201324
2810
03:36
I mean, look at me!
59
204158
1948
03:38
I mean, come on.
60
206625
1150
03:43
We had to first figure out
what my face actually looked like.
what my face actually looked like.
61
211320
5460
03:49
Not just a photograph or a 3-D scan,
62
217391
3027
03:52
but what it actually looked like
in any photograph,
in any photograph,
63
220442
3921
03:56
how light interacts with my skin.
64
224387
2460
03:59
Luckily for us, about three blocks away
from our Los Angeles studio
from our Los Angeles studio
65
227768
5249
04:05
is this place called ICT.
66
233041
2209
04:07
They're a research lab
67
235708
1272
04:09
that's associated with the University
of Southern California.
of Southern California.
68
237004
3403
04:12
They have a device there,
it's called the "light stage."
it's called the "light stage."
69
240871
3555
04:16
It has a zillion
individually controlled lights
individually controlled lights
70
244450
3714
04:20
and a whole bunch of cameras.
71
248188
1873
04:22
And with that, we can reconstruct my face
under a myriad of lighting conditions.
under a myriad of lighting conditions.
72
250085
6091
04:29
We even captured the blood flow
73
257589
1582
04:31
and how my face changes
when I make expressions.
when I make expressions.
74
259195
3092
04:35
This let us build a model of my face
that, quite frankly, is just amazing.
that, quite frankly, is just amazing.
75
263454
5260
04:41
It's got an unfortunate
level of detail, unfortunately.
level of detail, unfortunately.
76
269399
4333
04:45
(Laughter)
77
273756
1278
04:47
You can see every pore, every wrinkle.
78
275058
3696
04:50
But we had to have that.
79
278778
1600
04:52
Reality is all about detail.
80
280960
2381
04:55
And without it, you miss it.
81
283365
1867
04:58
We are far from done, though.
82
286793
1547
05:01
This let us build a model of my face
that looked like me.
that looked like me.
83
289363
3297
05:05
But it didn't really move like me.
84
293196
2746
05:08
And that's where
machine learning comes in.
machine learning comes in.
85
296871
2713
05:11
And machine learning needs a ton of data.
86
299608
3204
05:15
So I sat down in front of some
high-resolution motion-capturing device.
high-resolution motion-capturing device.
87
303497
4929
05:20
And also, we did this traditional
motion capture with markers.
motion capture with markers.
88
308450
4071
05:25
We created a whole bunch
of images of my face
of images of my face
89
313696
2976
05:28
and moving point clouds
that represented that shapes of my face.
that represented that shapes of my face.
90
316696
4087
05:33
Man, I made a lot of expressions,
91
321996
2811
05:36
I said different lines
in different emotional states ...
in different emotional states ...
92
324831
3460
05:40
We had to do a lot of capture with this.
93
328315
2667
05:43
Once we had this enormous amount of data,
94
331752
2891
05:46
we built and trained deep neural networks.
95
334667
3775
05:51
And when we were finished with that,
96
339117
1746
05:52
in 16 milliseconds,
97
340887
2444
05:55
the neural network can look at my image
98
343355
3112
05:58
and figure out everything about my face.
99
346491
2928
06:02
It can compute my expression,
my wrinkles, my blood flow --
my wrinkles, my blood flow --
100
350458
5476
06:07
even how my eyelashes move.
101
355958
2317
06:10
This is then rendered
and displayed up there
and displayed up there
102
358925
2835
06:13
with all the detail
that we captured previously.
that we captured previously.
103
361784
3222
06:18
We're far from done.
104
366077
1334
06:20
This is very much a work in progress.
105
368188
2207
06:22
This is actually the first time
we've shown it outside of our company.
we've shown it outside of our company.
106
370419
3321
06:25
And, you know, it doesn't look
as convincing as we want;
as convincing as we want;
107
373764
4194
06:29
I've got wires coming out
of the back of me,
of the back of me,
108
377982
2183
06:32
and there's a sixth-of-a-second delay
109
380189
2038
06:34
between when we capture the video
and we display it up there.
and we display it up there.
110
382251
4367
06:38
Sixth of a second -- that's crazy good!
111
386642
2816
06:41
But it's still why you're hearing
a bit of an echo and stuff.
a bit of an echo and stuff.
112
389911
3400
06:46
And you know, this machine learning
stuff is brand new to us,
stuff is brand new to us,
113
394104
3889
06:50
sometimes it's hard to convince
to do the right thing, you know?
to do the right thing, you know?
114
398017
4224
06:54
It goes a little sideways.
115
402265
2058
06:56
(Laughter)
116
404347
2422
06:59
But why did we do this?
117
407538
3229
07:03
Well, there's two reasons, really.
118
411077
2262
07:05
First of all, it is just crazy cool.
119
413363
2976
07:08
(Laughter)
120
416363
1008
07:09
How cool is it?
121
417395
1253
07:10
Well, with the push of a button,
122
418990
1992
07:13
I can deliver this talk
as a completely different character.
as a completely different character.
123
421006
4007
07:17
This is Elbor.
124
425823
2601
07:22
We put him together
to test how this would work
to test how this would work
125
430037
2312
07:24
with a different appearance.
126
432373
2135
07:27
And the cool thing about this technology
is that, while I've changed my character,
is that, while I've changed my character,
127
435450
4818
07:32
the performance is still all me.
128
440292
3273
07:35
I tend to talk out of the right
side of my mouth;
side of my mouth;
129
443589
2674
07:38
so does Elbor.
130
446287
1579
07:39
(Laughter)
131
447890
1150
07:42
Now, the second reason we did this,
and you can imagine,
and you can imagine,
132
450021
2790
07:44
is this is going to be great for film.
133
452835
2336
07:47
This is a brand new, exciting tool
134
455195
2701
07:49
for artists and directors
and storytellers.
and storytellers.
135
457920
4322
07:55
It's pretty obvious, right?
136
463131
1484
07:56
I mean, this is going to be
really neat to have.
really neat to have.
137
464639
2462
07:59
But also, now that we've built it,
138
467125
2055
08:01
it's clear that this
is going to go way beyond film.
is going to go way beyond film.
139
469204
2991
08:05
But wait.
140
473510
1150
08:07
Didn't I just change my identity
with the push of a button?
with the push of a button?
141
475653
3976
08:11
Isn't this like "deepfake"
and face-swapping
and face-swapping
142
479653
3033
08:14
that you guys may have heard of?
143
482710
1561
08:17
Well, yeah.
144
485208
1150
08:19
In fact, we are using
some of the same technology
some of the same technology
145
487074
2952
08:22
that deepfake is using.
146
490050
1600
08:23
Deepfake is 2-D and image based,
while ours is full 3-D
while ours is full 3-D
147
491954
4599
08:28
and way more powerful.
148
496577
2206
08:31
But they're very related.
149
499204
1666
08:33
And now I can hear you thinking,
150
501680
1889
08:35
"Darn it!
151
503593
1278
08:36
I though I could at least
trust and believe in video.
trust and believe in video.
152
504895
3801
08:40
If it was live video,
didn't it have to be true?"
didn't it have to be true?"
153
508720
2827
08:44
Well, we know that's not
really the case, right?
really the case, right?
154
512609
3522
08:48
Even without this, there are simple tricks
that you can do with video
that you can do with video
155
516727
3810
08:52
like how you frame a shot
156
520561
2776
08:55
that can make it really misrepresent
what's actually going on.
what's actually going on.
157
523361
4362
09:00
And I've been working
in visual effects for a long time,
in visual effects for a long time,
158
528263
3306
09:03
and I've known for a long time
159
531593
1932
09:05
that with enough effort,
we can fool anyone about anything.
we can fool anyone about anything.
160
533549
5226
09:11
What this stuff and deepfake is doing
161
539546
2388
09:13
is making it easier and more accessible
to manipulate video,
to manipulate video,
162
541958
4611
09:18
just like Photoshop did
for manipulating images, some time ago.
for manipulating images, some time ago.
163
546593
5371
09:25
I prefer to think about
164
553441
1298
09:26
how this technology could bring
humanity to other technology
humanity to other technology
165
554763
5036
09:31
and bring us all closer together.
166
559823
2294
09:34
Now that you've seen this,
167
562141
2359
09:36
think about the possibilities.
168
564524
1902
09:39
Right off the bat, you're going to see it
in live events and concerts, like this.
in live events and concerts, like this.
169
567810
4523
09:45
Digital celebrities, especially
with new projection technology,
with new projection technology,
170
573612
4735
09:50
are going to be just like the movies,
but alive and in real time.
but alive and in real time.
171
578371
3960
09:55
And new forms of communication are coming.
172
583609
2733
09:59
You can already interact
with DigiDoug in VR.
with DigiDoug in VR.
173
587088
4165
10:03
And it is eye-opening.
174
591699
2270
10:05
It's just like you and I
are in the same room,
are in the same room,
175
593993
3862
10:09
even though we may be miles apart.
176
597879
2373
10:12
Heck, the next time you make a video call,
177
600276
2841
10:15
you will be able to choose
the version of you
the version of you
178
603141
3736
10:18
you want people to see.
179
606901
1566
10:20
It's like really, really good makeup.
180
608974
2533
10:24
I was scanned about a year and a half ago.
181
612853
3579
10:29
I've aged.
182
617068
1721
10:30
DigiDoug hasn't.
183
618813
1650
10:32
On video calls, I never have to grow old.
184
620798
3492
10:38
And as you can imagine,
this is going to be used
this is going to be used
185
626322
3088
10:41
to give virtual assistants
a body and a face.
a body and a face.
186
629434
3213
10:44
A humanity.
187
632671
1192
10:45
I already love it that when I talk
to virtual assistants,
to virtual assistants,
188
633887
2762
10:48
they answer back in a soothing,
humanlike voice.
humanlike voice.
189
636673
2933
10:51
Now they'll have a face.
190
639919
1776
10:53
And you'll get all the nonverbal cues
that make communication so much easier.
that make communication so much easier.
191
641719
4882
11:00
It's going to be really nice.
192
648171
1420
11:01
You'll be able to tell when
a virtual assistant is busy or confused
a virtual assistant is busy or confused
193
649615
3637
11:05
or concerned about something.
194
653276
2680
11:09
Now, I couldn't leave the stage
195
657694
2509
11:12
without you actually being able
to see my real face,
to see my real face,
196
660227
2698
11:14
so you can do some comparison.
197
662949
1684
11:18
So let me take off my helmet here.
198
666573
1876
11:20
Yeah, don't worry,
it looks way worse than it feels.
it looks way worse than it feels.
199
668473
4770
11:25
(Laughter)
200
673267
2428
11:29
So this is where we are.
201
677188
1698
11:30
Let me put this back on here.
202
678910
1604
11:32
(Laughter)
203
680538
1950
11:35
Doink!
204
683115
1186
11:37
So this is where we are.
205
685292
1600
11:39
We're on the cusp of being able
to interact with digital humans
to interact with digital humans
206
687997
3701
11:43
that are strikingly real,
207
691722
2181
11:45
whether they're being controlled
by a person or a machine.
by a person or a machine.
208
693927
3269
11:49
And like all new technology these days,
209
697220
4375
11:54
it's going to come with some
serious and real concerns
serious and real concerns
210
702531
4746
11:59
that we have to deal with.
211
707301
1734
12:02
But I am just so really excited
212
710017
2118
12:04
about the ability to bring something
that I've seen only in science fiction
that I've seen only in science fiction
213
712159
5053
12:09
for my entire life
214
717236
2270
12:11
into reality.
215
719530
1328
12:13
Communicating with computers
will be like talking to a friend.
will be like talking to a friend.
216
721752
4222
12:18
And talking to faraway friends
217
726323
2500
12:20
will be like sitting with them
together in the same room.
together in the same room.
218
728847
3273
12:24
Thank you very much.
219
732974
1308
12:26
(Applause)
220
734306
6713
ABOUT THE SPEAKER
Doug Roble - Computer graphics software researcherDoug Roble has found a career combining the things he loves: math, computers, movies and imagination.
Why you should listen
Doug Roble has really only had one job in his life. After getting his PhD in Computer Science from the Ohio State University in 1992, he joined Digital Domain, a visual effects production company. Once there, he found a unique place where art and technology collide. Now he builds new tools for artists to use and they, in turn, use the tools in surprising and unexpected ways. The feedback loop between art and science is completely addicting. And, the byproduct of this are movies that the whole world enjoys.
Roble's work outside Digital Domain reflects this passion. He was the Editor and Chief of the Journal of Graphics tools for more than five years. He's currently the Chair of the Motion Picture Academy's Sci/Tech Awards and a member of the Academy's Sci/Tech Council. And two of the tools he's built over the years have won Sci/Tech Academy Awards themselves.
Doug Roble | Speaker | TED.com