Our First Reddit Bot - Coloring B&W Photos Using AI Deep Neural Network Machine

by Amir Avni

EDIT 9/8/16: The bot has been working for two weeks now and we've created some gallery and statistics. Also, some videos examples, Check it out.

EDIT 24/7/16: The bot works on r/all, which means it works almost anywhere on Reddit. Also, it works when it gets a private message.

As writers on this blog and as the technology hobbyists we are, we always share a to-do list with many projects we want to do in the future. There's always so little time for our hobbies and the list just getting bigger and bigger. One of the old projects on this list was creating a Reddit bot. What is a Reddit bot? It is a program which scans posts/comments on Reddit, and can react to the information it collects. Our first thought was to create a project in the data analysis field. I really love posts such as how many comments it takes in a discussion until someone reminds Hitler and others.

Last weekend I bumped into this website and paper by Richard Zhang, which trained an AI program using deep neural network to receive as input a B&W photo and outputs a colored photo. I was already really amazed by the deep learning latest developments and decided to go deeper into this subject.

Another unrelated fact, is that I'm a big fan of old photos. I follow the sub-Reddit r/oldSchoolCool for a long time now, checking the old photos people are uploading, imagining the stories behind the photos and reading the comments.

So, since I'm a big fan of these photos and wanted to start sniffing around the issue of Deep Learning and wanted to build a Reddit bot, I decided to build a Reddit bot which will react to people requests to color their B&W images. Notamacuser joined me and we built a bot which respond to "colorizebot" requests as follows:

  • Downloads their image.
  • Enhance the image a bit.
  • Pass it through the neural network.
  • Upload the result to imgur.com and gets the imgur link.
  • Reply to the request on Reddit with the link.

In this post I'll describe shortly the entire process, and share our code.

Check the Imgur gallery to see some examples.

What are Deep Neural Networks and How to Use them?

What?

Not going too much into mathematics, neural networks are a set of a lot of "digital neurons" where each neuron reacts differently to a different combination of inputs. We can think about a very small network with several neurons for example, where each neuron sums the amount of a certain color in a picture. The first neuron reacts only to the red layer of an RGB picture, the second receives the green layer and so on. An output of this kind of network can be an answer to a question "What is the strongest color in a given picture?". Now to the interesting fact, in these kind of networks we do not define the reaction of a single neuron (say, to red pixels), instead we let it learn the facts by itself using a training process. We can do so for example by inputting a lot of strong red pictures and on the output let the network knows the answer is "red". By doing so we are "training" the network where eventually this neuron will know to strongly react only to strong red color.

Let's go much larger scale now - Deep neural network consist of million of neurons (These days) in several layers of neural networks. The input to the first layer in our case is the RGB picture, but the input to the second layer is the output of the first layer, and our final result is the output of the last layer.

That way we can build much more complex systems which when it comes to photos, attempt to mimic the way the human brain works. Comparing it to our visual system, our brain is also built as a several layers network as much as we understand today. The first layer perceiving the image and might react to lines in a image, the second could react to some curves, and so on, a deeper layer will already detect faces and objects and will makes us to react to these people/objects.

Which?

This was a very simplified explanation about neural networks, but the fact is that amazing stuff can be done with these network, just look for the thousands of examples on Google (Many of Google features are based on deep neural networks as well by the way). I chose to use an already trained network from this site. The network can color black and white images, here's an example of the first picture I found on Google searching the words "New Zealand" (Left - Original, Middle - After removing colors, Right - Restoring the colors):

Eventually this is only a machine, and this is still a very new idea, so the results aren't always that good:

The network was trained using hundred of thousand of photos, and my intuition can tell me that the second photo came out very different from the original since the original colors most probably were not represented in many photos in the training set. With old B&W photos it is even harder, since the network has never been trained with these type of photos (In terms of image quality, and grey-scale differences). Nevertheless, it seems to do a good job sometimes, so it was time to download the code and use it.

How?

In order to use this network on a windows OS (as I use), you first need to download CAFFE, a framework for deep neural networks. If you are not too technical you might have some difficulties. I followed the steps on their Readme file and it went OK.

  • Download the environment.
  • Install Visual Studio 13.
  • In case you don't use GPU, change parameters in .\windows\CommonSettings.props to CpuOnlyBuild
  • Set PythonSupport to true (In case you are using Python)
  • Define your %PYTHONPATH% under windows environment.
  • Update you python libraries: numpy scipy matplotlib scikit-image protobuf opencv (opencv is needed only for our part of the code, not the original one)
  • Compile the code.
  • Move the pycaffe compiled code to the the /lib/site-packages/ folder in your Python directory.

Now you are ready to use the code which supports the "colorizing" neural network. In their website there are some examples for how to use it. You can also use our code in order to take a folder of B&W images and restore the colors of all the images.

Here are some examples I made from photos on r/OldSchoolCool:

Of course not everything is perfect:

In my code, I did a few more transformations before sending the picture to the neural network:

  • Stretching the histogram.
  • Transform the image to LAB colors, and subtracting the mean of ImageNet L-value from each pixel (This is the original set of images which was used in the training operation)

This is the place to ask image processing people and other photo lovers - What would YOU add to the pre-process part in order to make the old B&W photo to resemble more to new B&W photos? Even if you not into coding, just like playing with Photoshop, write your thoughts on the comments section.

Reddit Bot

By now, we have a Python function which colorizes photos using CAFFE. Next we need to build a bot which will communicate with the users on Reddit. In order to build the Reddit bot we used Praw library which is a python extension that helps creating Reddit crawlers and bots. There is a simple example on their website for people who want to experiment with this.

Before start using this library, you must open an app on Reddit, read carefully the terms of service so you won't get banned. Basically you should state there what you are planning to do on this app, and as long you don't need users to use this app, you can do almost anything.

Our code can be found here, and I believe it is not so hard to understand. Start by checking bot_main.py.

  • We login to Reddit, and start asking for a stream of comments from a selected sub-Reddit.
  • When we find the word "colorizebot" in one of the comments, we go to the action_bot function, passing all the information of the comment, thread, user etc.
  • Then, we check the attachment of the original poster, and if it is an image we download it, and send it to the neural network.

  • When we get the output, we save it to an image file, and upload it to Imgur.com. In order to upload images we used the Imgur API. Also here we had to create an imgur application so we can upload photos freely through python. To upload the image, we used the Imgur python library (because we used python to create this bot).

  • After uploading the image successfully, we finish the action_bot function by replying to the comment with the link on IMGUR. Then we go back to listening to the stream of comments.

To conclude, or if you just scan the photos on this post and got here, our code is on Github, In this library you can also find some simple bots named : most_used_words_subreddit.py and simple_image_download_bot.py .

This was our first encounters with deep neural networks and Reddit bots. If you have an idea on how to make our process better, we would love to hear, leave a comment.

On the meantime, check r/OldSchoolCool and ask to "colorizebot" images as a comment, or just head to r/colorize_bw_photos, to test this bot, upload an image and comment "colorizebot". Most important, don't forget to up-vote our replay :)

AA

P.S

As a bonus - Our blog main photo, finally got some colors: