AI Makes Stunning Photos From Your Drawings (pix2pix) | Two Minute Papers #133

AI Makes Stunning Photos From Your Drawings (pix2pix) | Two Minute Papers #133

Dear Fellow Scholars, this is Two Minute Papers
with Károly Zsolnai-Fehér. In an earlier work, we were able to change
a photo of an already existing design according to our taste. That was absolutely amazing. But now, hold onto your papers and have a
look at this! Because here, we can create something out
of thin air! The input in this problem formulation is an
image, and the output is an image of a different kind. Let’s call this process image translation. It is translation in a sense, that for instance,
we can add an aerial view of a city as an input, and get the map of this city as an
output. Or, we can draw the silhouette of a handbag,
and have it translated to an actual, real-looking object. And we can go even crazier, for instance,
day to night conversion of a photograph is also possible. What an incredible idea, and look at the quality
of the execution. Ice cream for my eyes. And, as always, please don’t think of this
algorithm as the end of the road – like all papers, this is a stepping stone, and a few
more works down the line, the kinks will be fixed, and the output quality is going to
be vastly improved. The technique uses a conditional adversarial
network to accomplish this. This works the following way: there is a generative
neural network that creates new images all day, and a discriminator network is also available
all day to judge whether these images look natural or not. During this process, the generator network
learns to draw more realistic images, and the discriminator network learns to tell fake
images from real ones. If they train together for long enough, they
will be able to reliably create these image translations for a large set of different
scenarios. There are two key differences that make this
piece of work stand out from the classical generative adversarial networks:
One – both neural networks have the opportunity to look at the before and after images. Normally we restrict the problem to only looking
at the after images, the final results. And two – instead of only positive, both positive
and negative examples are generated. This means that the generator network is also
asked to create really bad images on purpose so that the discriminator network can more
reliably learn the distinction between flippant attempts and quality craftsmanship. Another great selling point here is that we
don’t need several different algorithms for each of the cases, the same generic approach
is used for all the maps and photographs, the only thing that is different is the training
data. Twitter has blown up with fun experiments,
most of them include cute drawings ending up as horrifying looking cats. As the title of the video says, the results
are always going to be stunning, but sometimes, a different kind of stunning than we’d expect. It’s so delightful to see that people are
having a great time with this technique and it is always a great choice to put out such
a work for a wide audience to play with. And if you got excited for this project, there
are tons, and I mean tons of links in the video description, including one to the source
code of the project, so make sure to have a look and read up some more on the topic,
there’s going to be lots of fun to be had! You can also try it for yourself, there is
a link to an online demo in the description and if you post your results in the comments
section, I guarantee there will be some amusing discussions. I feel that soon, a new era of video games
and movies will dawn where most of the digital models are drawn by computers. As automation and mass-producing is a standard
in many industries nowadays, we’ll surely be hearing people going: “Do you remember
the good old times when video games were handcrafted? Man, those were the days!”. If you enjoyed this episode, make sure to
subscribe to the series, we try our best to put out two of these videos per week. We would be happy to have have join our growing
club of Fellow Scholars and be a part of our journey to the world of incredible research
works such as this one. Thanks for watching and for your generous
support, and I’ll see you next time!

Dereck Turner

86 thoughts on “AI Makes Stunning Photos From Your Drawings (pix2pix) | Two Minute Papers #133

  1. Two Minute Papers says:

    Make sure to post your results here for our amusement! Or in case you find some interesting results elsewhere, links are also welcome. 🙂

  2. Rob ʺEuphoricAgnosticʺ McDoritos says:


  3. Y H says:

    This is interesting. Just under a week ago DeepMind published PathNet which displayed proto-General Intelligence qualities even if it just on Atari games. Few videos ago you mentioned similar paper, but this paper shows a waaaaay much better capability on image generation.
    Imagine if some research group combine those two.

  4. Ismael Goldsteck says:

    this is one of my favorite, if not the favorite channel on YouTube

  5. Eduardo Soto says:

    Ice cream for my eyes. <3

  6. 噢馬 says:

    "Do you remember the time when video games where handcrafted?"
    Every time I watch your videos or see new occurrences in the field,
    I think about this without using these concrete words.

  7. Paallikko says:

    Subscribed at 11000, I like you more after every new video 🙂

  8. Quenz says:

    "Ice cream for my eyes!" 😀

  9. Dave Jacob says:

    this is simply awesome.

  10. CGPacifica says:

    And here I've been thinking "Well at least creative jobs will be safe when autonomy takes over the rest of the jobs…" FML

  11. Winding Light says:

    If they get something like this to work for videos, then it would be difficult for people to determine what's real and what's fake. People have already been fooled by photoshopped images. How long until people start manufacturing "evidence" for courts or generating "fake news" footage?

  12. BjarkeDuDe says:

    Khajiit is not entertained by your shenanigans

  13. Bishshoy Das says:

    Fucking mind blown.

  14. Yongliang Qin says:

    just read this a month ago

  15. allinonemovie says:

    Considering the amount of work you have to do for each video, it's incredible how many videos you upload. Just two words: Thank you!

  16. m ・ ́ω・ says:

    Isn't this how human imagination works? Are we on the dawn of creating thinking machines?

  17. Mr Tomato says:

    love your videos keep uploading

  18. Eyesonly - 目だけ says:

    Been here since 1k subs. Your content is amazing. Thanks for the video

  19. Shaul Kedem says:

    What is this cat on the first slide, I thought we got to that level of generative images 🙁

  20. Hyunsung Go says:

    Have you ever heard of HTM(Hierarchical Temporal Memory)?
    It's kind of like a neural network but much better.
    The creators of HTM tried to mimic the neocortex and it works really well.
    They claim that it's the real way of true intelligence.
    I just think it's incredibly awesome and I don't think it doesn't get much attention it deserves.
    After all, isn't the neocortex that makes human intelligent, right?
    p.s. Sorry for my bad English ;(

  21. Japan is Sinking says:

    So I was sitting in Spanish class the other day when I got an idea, and I want another take on it.
    You know how sites like Google translate always churn out characteristically faulty and unreliable translations? Well, would it be possible to improve machine translations with the aid of a general AI?
    How I imagine it would work would be similar to the other general AIs discussed on this channel. It would start out with the broken machine translation first, make its evolved changes to it, and then see how close it is to the same sentence translated by a professional translator (there are hundreds of books translated from English to Spanish every year, so a large training data set wouldn't be that hard to acquire I imagine.) The closer the edited bad translation is to the proper translation, the more fitness points the AI is rewarded with.
    Would this even work, or does language have too many subtle complexities for human-made code to be improved upon? I watch the papers featured here and especially in the image generation ones the neural networks' understanding of how RGB values come together to make recognizable images seems impossibly deep for a computer, surely this same understanding can be reached in regards to language? Or maybe I'm missing something in my ignorance?
    Anyway, fantastic channel man, it isn't often that someone finds such a small niche and pours so much effort into filling it!

  22. krzysztofnatalicz says:

    You can try it as a app now.

  23. dewinmoonl says:

    cool recap. Isola actually came to MIT yesterday to give a talk, it was cool 😀

  24. apolotary says:

    Time to open an artisanal handcrafted game studio

  25. Christopher says:

    I like two minute papers, but some of these papers may end up strictly becoming fun apps on smart phones; never truly lifting off as with the case of VR.

  26. VoidMoth says:

    they look like they about to solve the riemann hypothesis, and I look like Ive just figured out how to draw phallic objects in the sand, by writing a temperature plotting thing. awesome vid tho

  27. Pacdev says:

    4 minutes paper

  28. SUV Tropics says:

    I wonder if there will be a human massacre run by robots.

  29. TedRobotBuilder says:

    It it works for video/animation, this will be huge. I don't see why it wouldn't.

  30. Hristo Vrigazov says:

    Best channel ever

  31. Craig Wall says:

    I'd hate it if AI replaced artists and designers.

  32. Vinay Seth says:

    Just tell me when I'm going to become obsolete :/

  33. Marius Langeland says:

    Why are we still here, just to suffer?

  34. Twistymoo says:

    This ruins the purpose of actual artists.

  35. AlphaCore says:

    this is my new favorite channel

  36. Chris Walsh says:

    ice cream for my eyes LOL!

  37. Overkin says:

    we'll soon be able to "translate" stick-figure drawings to porn. The golden age of erotica!

  38. Constant Throwing says:

    You computer people are smart fellers.

  39. ksztyrix says:

    Those are not photographs, but simulacrums.

  40. Bipin Oli says:

    Where do you find all these information about the latest papers?

  41. Alex Taylor Barratt says:

    I love your channel so much!!!!!!

  42. Kade Blad says:

    This is amazing!
    And it's written in Python!!! 😀

    SO COOL!!!!!!!!!!!!!!!!!!

  43. Ivan Damico says:

    Thanks for the evaluation AND for supplying the links! I'm already on Github and have set up my account and learning code branches and repositories (sounds like a bank!) and as the process of learning and sharing information is growing exponentially, my hats off to you for sharing the latest of your discoveries and insights, and not waiting until your patent comes through to share what you have learned like so many others do. To create a new method is inspired, to share with the world, divine.

  44. Bannicus says:

    Maybe it could be used for mouth movements in animation so eg. Pixar can dub content in each language and have the software animate mouths and have them work properly in all languages.

  45. Player_1 says:

    We need 3D models of those horrifying animals

  46. Adrian Salamunovic says:

    What does "ground truth" mean between input and output?

  47. S X says:

    You should really start to give credit to the actual authors and keep the original title.

  48. Swift Fox says:

    "Bespoke, artisinal hand-crafted games". I can imagine somebody using this as their sales pitch.

  49. Frank Anzalone says:

    Four and a half minute paper

  50. Limitless 1 says:

    wow this is soo cool
    i just played with the website
    thanks 🙂

  51. LORDE 2729 says:

    lol i like how he says his name. "karow zohor…..blaballab"

  52. Codex Group says:

    Is there an app or service that will let us summarize documents using AI? Like a two minute summary of a PDF?

  53. Solve Everything says:

    Maybe this can be used to turn lowrez images int high rez images? And make resolution in games go up?

  54. ello propello says:

    i am not able to find this tool. no link i found so far contained a working online demonstration

  55. 4Dm8ion says:

    Wish I could UL my drawings of my cat instead of redrawing w a mouse.

  56. Isai Karnadhi says:

    This is how the Chinese designed knockoffs…

  57. AnteConfig says:

    imagine video game graphics like peoples faces in games being drawn in this fashion. That would be soo good.
    Or cartoons and if the voices are artificially crafted as well the cartoons can run for decades without needing to higher voice actors.
    Imagine TV shows like The Expanse or Stargate but with no actors.
    Why use your voice on the radio when you can have a machine generate another one.
    this is great stuff.

  58. Irene Plaster says:

    Wheres the link to make your own? why do you refuse to post the link for that?

  59. adfdasfadfdaaaa says:

    "Open the pod bay doors, HAL!" This is borderline terrifying to me. I don't want to sound backward minded, but I can see no way all this progress wouldn't go wrong. Society is not flexible enough and neither smart enough (as a whole) to keep the pace and adapt to AI, which will soon be able to evolve based on it's own decisions. The tool will become smarter than the user. Change my mind. Please.

  60. Sancarn says:

    Hello World

  61. Leo Zendo says:

    What is the cat for?

  62. luis pacheco says:

    pix2pix: Turning new products into used products

  63. Zorn101 says:


  64. YS says:

    AI is still way far from artistic QC, direction and control. Needs more development..

  65. Yves Gomes says:

    I'm playing with it. My first result was terrible, probably mostly because I tried to draw the whiskers.

  66. Surekha Sarode says:

    Its really awesome!!!
    Can it be used for human beings, so that it can be helpful for the crime branch to identify criminals by the sketch drawn?

  67. Gustavo Martinez says:

    How can I become a patreon? I just have cash!!!, no card!!

  68. DunnickFayuro says:

    At one point in the future, this sorts of AI will be embeded into graphics renderers of games.

  69. videolabguy says:

    WHEN the AI takes over and conquers the human race, we'll be lucky to be kept as pets instead of "a bundle of raw materials". This is a bad path to follow. I just hope the AI doesn't have this narrators horribly irritating voice.

  70. ChristianIce says:

    Yeah, tried the demo, doesn't do anything.
    You draw what you are supposed to, it elaborates and then a white rectangle appears.

  71. Rainbow Doodler209 says:

    This is very very impressive, but AI hopefully won't rule the world. Remember, our robots are not like skynet.

  72. Le Wang says:

    Anyone able to generate good-looking picture from drawing on that website? For me, it looks terrible.

  73. Jason Hanson says:

    Won't be for a long time. It looks like crap had a baby with a cat.

  74. Justin Saephan says:

    looks like a bad dream or some chinese knockoff.

  75. raintz randmaa says:

    With these, they can start making fake news, in other words, humans are gone soon…

  76. Beedy KH says:

    Oh nice. Now governments can fake satellite images and hide secret projects.

  77. Thomas Dietrich says:

    This is kinda just pix2pix but if it was better.

  78. zillBoy says:


  79. José Leonardo Diaz Ordoñez says:

    2019 -> Deep nudes.

  80. Jeff Greenwade says:

    AI will be able to automate creative jobs much faster than people think. We need to get ahead of the transformation before drastic changes cause damage in society. Andrew Yang is the only candidate running right now who is informed on automation and its effects.

  81. Dim tass says:

    Next step for entertainment AI is to put yourself and friends as the main character in movie in real-time.

  82. Artisan says:

    This is just like the photoshop tool that clones a texture, the healing brush, but more advanced. I mean, this is not "artists died", it's more like "great, we artists can design more and better things in less time". It's just a tool.

  83. Artisan says:

    Video games were never hand crafted, the computer was always there and most of the textures were generic and tiled textures. This will be more more close to hand crafted than the old games if you consider the old games somehow hand crafted. Also, there is things that doesn't exist so artists will be creating those things. Then the price factor, you might have the technology but if most studios is just cheaper to so something else even with many artists working on it, that will be better.

    For me in the end this will be like a book. The artists will be the writer. The book will be the story generated with images, a 3d game, a world full of life, but still a narrative of that world will be there and artist driven. In my opinion Machines will never replace artists because IA is not really possible and even if it was at that time no fucking one will need to work at that point so i'ts irrelevant, Africa would be rich like the rest of the world, if not that means artists are still needed for political reasons.

    We are 200 years later saying "machines will replace humans, be scared" this started even before French revolution and nothing happened, in the end machines are just tools to speed up things or make them better.

  84. TURRO27 says:


  85. Tiju John says:

    20:40 generated images feels like what you recall from a dream

Leave a Reply

Your email address will not be published. Required fields are marked *