Family fun with deepfakes. Or how I got my wife onto the Tonight Show

[ Update 3 Feb 2018: added two new creations at the bottom of this post. Last one turned out really well ]

(Images might take a while to load!)

I’ve first heard of deepfakes a good week ago. Thanks Twitter. Thanks Tim Soret.

There’s something called Deepfakes on internet, and it’s the most cyberpunk shit you can imagine. Machine learning used to swap porn actresses faces with hollywood stars. Obviously NFSW.

 — @timsoret

Yes, it’s pretty damn cyberpunk. But from a superficial point of view, /r/deepfakes (extremely NSFW! You have been warned) consists of people using an app created by user “deepfakes” to create fake celebrity porn.

This has caused a shitstorm on the Internet, media discussing the legality of it all, websites taking down the deepfake creations, and people panicking as they realise AI is going to screw us all up (newsflash: it’s already been happening in much less obvious ways). And meanwhile, Nicolas Cage is taking over Hollywood.

While everyone’s debating whether this is good or bad, I just had to find out more. First thing that came into my mind? How can I apply this to everyone I know (in a non-porn way, in case you wondered).

How does it work?

The deepfakes app is a deep learning algorithm that learns how to reconstruct faces. Give it a bunch of pictures, let it run a few hours, and it spits out fuzzy copies of those images. Do note, it doesn’t create a copy. It learns what a face looks like, in different expressions, and is able to output that face solely based on that. There’s a detailed explanation on Reddit but let me try and dumb it down.

Think of it like this: imagine if you could stare at someone for 12 hours straight, observing all their expressions and absorbing that into your brain. Then that person asks you to sketch his face on paper, smiling, crying, any expression you’ve observed. What do you do? You immediately generate a photographic quality sketch on paper, from the mind! (using a pencil)

It’s insane!

While that’s pretty cool, it only gets better. See that “encoder” part? The FakeApp uses one encoder for all faces. It’s the decoder that’s kept face specific. And here comes the really cool part. Let it learn two faces, and things become more interesting.

Right, now see how this works. The encoder takes an image of a face, let’s it run through its “brain”, and the decoder spits it out again. In the example above, it can do so with faces of Anne Hathaway, and Elke, my wife. Ok, so far so good. But now let’s feed it a picture of Anne, but use the decoder that generates Elke’s face!

You just created a new photo of Elke. A photo that never existed, in the same angle, the same expression, as Anne! Amazing!

Family fun

Sure, putting celebrities’ faces on your favorite porn stars is an interesting use case. But we can leverage these celebrities for other things, such as inserting your friends and family into blockbuster movies and shows!

For the best result, you must first find an actor/actress that has a similar head shape as the person you wish to insert. In case of Elke (my wife) I accidentally noticed, while watching the Dark Knight Rises, that Anne Hathaway might be a good fit. I guess you know Anne, so here’s Elke:

All I needed was about 850 photos of Elke, a few 1000 of Anne, a lot of computing time, et voila: Elke’s on the Tonight Show starring Jimmy Fallon.

Bonus effect: now we know what Elke looks like with short hair :D

Here’s a little comparison gif:

I personally think it’s fun, can be innocent, and even makes for a nice surprise/gift. Remember, any tool can be used for evil. And as long as we’re not banning guns, this should not be a high priority, amirite?

There’s so much you can do with this technology. You know those dumb emails people send around where they replaced dancing elves heads with their own, or even worse, yours? Well, now you can put your best friend into his favourite movie: have her dance with Patrick Swayze and have the time of her life, or have an alien burst out of his stomach. It’s all within your reach now!

Beyond just pure fun, I can only imagine how people will start turning this tech into business ideas. Fashion will be huge (what would I look like with this kind of hair, this kind of dress…), fitness could be interesting (do I look good with muscles, will I really look better skinny), travel (this is you standing on a beach is going to be quite convincing). It’ll bring advertising to a whole new level. No need to imagine what if, they’ll tell you what your “better” life will look like! And it’ll be hard to get that picture out of your head…

Update: in the mean time, I’ve created two more. Elke’s a huge fan of Steve Carell, and I suddenly realized Anne Hathaway co-starts with him in Get Smart. First attempt was okay:

Then I wanted to try this one (original video):

And I think it turned out great:

Originally published at sven charleer.

Family fun with deepfakes. Or how I got my wife onto the Tonight Show was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.