Lost Interview about Style Networks
and the Deep Forger Bot

Blog / Lost Interview about Style Networks
and the Deep Forger Bot

How does Deep Forger generate its paintings?

Well, Deep Forger is made up of three parts, and only one of them is a deep neural network. There are many neural networks available for things like image recognition, and the one behind Deep Forger is cutting-edge but now widely used: it’s called VGG and has 19 layers, you can even download it in a 574Mb file! Sitting on top of this neural network is an optimization algorithm based on a research paper called A Neural Algorithm of Artistic Style — which allows you to combine different features from two images and generate a third output image. (This was published less than two weeks ago!)

The Deep Forger was the first bot to make this algorithm available to non-programmers. To make it more accessible it has some extra components that make it particularly interesting and easy to use! One of these components is a library of thousands of famous paintings, serving as a reference when forging the user’s photos; Deep Forger has some understanding of the paintings and will try to find those that best match the content. It achieves this via an AI Curator that helps find the best painting and parameters, then a tongue-in-cheek AI Critic that gives its assessment of the quality before posting. Both of these AIs are learning from the feedback received on Twitter (Favs or Retweets); it’s still early days but results are getting visibly better.

What are Deep Forger’s capabilities?

The idea is to lower the barrier of entry for non-programmers and make it as simple as possible to commission Generative Art: you send the bot your photo along with a description of what you want, and it’ll return a painting that it generated! By default, it will pick randomly one of the famous painting it knows from its database, but you can also use commands like “!sketch” “!stylized” or “!abstract” to commission a particular type of painting.

We let users also pick which painters they want to see using keywords, so “+Cassatt” (to add Mary) or “-Picasso” (to remove Pablo) will filter the list accordingly. So far, users have created new avatars or background images, or simply posting cat pictures and internet memes—obviously!

How is the system trained to generate paintings?

The “teaching” can be broken down into three parts again, and was the combined effort of many researchers and developers in the field. First comes the training of a deep neural network (“VGG” as mentioned previously) on an image recognition problem. This takes a very long time and the best GPUs you can buy, but it’s now a relatively standard optimization process and you can even download pre-trained versions. The second process is the algorithm that generate an image from scratch (based on the Neural Style[2] paper): using features found by the neural network, it optimizes your photo (changing the pixel colors) to keep its content but also match the style of another image. Third, there’s a component that selects the right image, scale and parameters to match your photo and help generate good results. This is done by an algorithm that compares features of the image in the database and matching them with the features in the image—for example the color scheme. This is where the social network comes in; the code gathers statistics from Twitter that it uses to influence the algorithm and adapt over time.

What was the inception of the art project?

The whole machine learning (ML) community is booming right now and the field combining ML with generative art is just emerging! A few months ago it was Deep Dreaming, then applications in Recurrent Neural Networks, but with the recent publication of the “Neural Style” there’s more potential for the tools to be helpful for artists—and usable by general users too. As soon as the paper was published, I knew there’d be an army of hobbyists and professionals working to replicate the results. And indeed, within a week, there have been almost half a dozen, in particular the one by Kai Sheng Tai (first available) and by Justin Johnson (powering Deep Forger).

Even before the code was available, I speculated what the missing components would be and how I could best contribute to the field. Putting up a bot on Twitter generating near-HD images seems an obvious step, as it would help the community quickly explore the medium and figure out what the technology is capable of. The idea of gathering an encyclopedic database of art was a good starting point, and is turning into a great platform for helping understand what “Style” actually is and how you can explain the style of various painters. For example, it’s possible to gather the common style (or average of all styles) used by Picasso in all his paintings, and apply that to your photos! Getting feedback on Twitter also will help the bot understand which styles are preferred by followers and how they combine with specific content. There’s really a lot to do here—and we’re only just getting started :-)

What are, to you, fascinating capabilities of current AI?

The most promising idea is that neural networks may help artists be more creative or productive, for example by helping fill in the gaps and perform repetitive work, or coming up with creative suggestions. For example, multiple game developers have been using Deep Forger to try out Art Style experiments and see possible styles/elements that they might not have thought of!

Another very interesting aspect is that machine learning can eventually capture, for example, the craft of Rembrandt or Mary Cassatt: the grains of their canvas, the strokes of the paint, their common elements, and aspects of the painting composition. Neural Networks cannot replicate the Artist, but we may be able to get an additional, unique understanding of Fine Arts thanks to these techniques. It’s fascinating for a variety of reasons — not least copyright — but that’s a discussion for another time!