It is the beginning of a new month, so it’s time again for me to write a summary about what we have been doing at Helsinki Rephotography in September. Last month was the Hack4OpenGLAM event, so most of our time went to preparing for that. We were participating with a project about upscaling images with machine learning and artificial neural networks. For this we used the OpenCV library which stands for Open Computer Vision.

Artificial neural networks are computing systems that are inspired by the real biological neural networks found in our brains. They can be used for example in pattern recognition, 3D reconstruction and machine translation. For testing, we used a neural network model trained for upscaling images called EDSR.

Setting everything up

I was the one responsible for doing the initial legwork and getting OpenCV up and running. This required compiling it from the source code, since the super resolution module necessary for upscaling images was not included in the pre-compiled version of OpenCV. I first did a test version without GPU acceleration just to see how fast the upscaling was with just the CPU power. This ended up working with simpler upscaling models such as FSRCNN, but that one obviously wasn’t as good as EDSR, which took well over a minute to upscale a small image with just the CPU.

I then figured out how to get CUDA working with OpenCV. This allowed the use of Nvidia GPUs in the upscaling process. It required installing the CUDA toolkit and recompiling OpenCV with support for it. After setting it up, I ran into another problem: my GPU had too little integrated memory to do the upscaling. This prompted Kimmo to set up an analytics server with a more powerful GPU that we could try, and this ended up working. It now only took 5 seconds to upscale the same picture that took over a minute just with the CPU.

The results

After this we could try it on different images and see how it worked. We ended up using 3 different image upscaling models; GFPGAN, Real-ESRGAN and ESRGAN. These models were based on a different neural network technique called generative adversarial network. In this technique, two neural networks compete against each other. The other creates data points, while the other tries to distinguish between the real ones and the ones created by the neural network.

The end result was that the images themselves looked good, but the AI sometimes tended to create weird things, such as badly shaven mustaches, faces inside faces and walls that were made from fabric. So, it was not perfect, but could work if you weren’t looking for the perfect result. Here is a good example of the upscaling that we did on an image provided to us by Fortepan, who we did this project in collaboration with. This image was upscaled with ESRGAN. In it you can clearly see the difference in sharpness and details between the original and the upscaled image.

For me it was interesting to see and learn what machine learning and neural networks can be used for. I hope that in the future we can also use this for something useful.

-Siru, intern

This month in Helsinki Rephotography we have worked on implementing IIIF using IIIPImage to Ajapaik. IIIF stands for the International Image Interoperability Framework, and it is a protocol for standardizing image retrieval. Using IIIF it is easy for users to manipulate images, like modifying size, scale, region of interest, rotation, quality, and format. IIIF also allows for easy implementation of image annotations, and with it users can comment, transcribe and draw on images.

The IIIF protocol has two parts, the image API, and the presentation API. The image API delivers images in a way that easily specifies the characteristics of the image, such as region, size, rotation, and quality. The presentation API on the other hand provides the information necessary to drive a compelling viewer experience.

IIPImage is a high-performance image server that can be used to stream high resolution images. It uses IIIF and the Fast CGI protocol in combination with a front-end web server (nginx in Ajapaik’s case). IIPImage can use images that are either in the TIFF format, or the JPEG2000 format. It also comes with an image converting tool called vips than can be used to convert images to the TIFF format.

During the month I have made a Python script that is used with the uWSGI server to download images from Wikimedia, convert them to the TIFF format and then display them over IIIF. uWSGI is a Web Server Gateway Interface server implementation that can be used in conjunction with a web server to forward requests to an external program.

The idea is to get this script to execute whenever a 404 Not Found error happens at the IIPImage server. It will parse the filename from the request and find out if the image can be found on Wikimedia Commons. If it can, it will download and convert the image to TIFF using vips. Lastly it will redirect the user to the IIPImage server that will serve the image using the IIIF protocol.

I did this by launching uWSGI with the Python script and a socket file as parameters. I then added a new location to the nginx site configuration that specifies that the requests to this location should be forwarded to the uWSGI server. This worked when the user directly accessed the location but didn’t when they were forwarded to it from a 404 error, since in that case only the request URL got forwarded to uWSGI instead of the full parameters.

To fix this we had to modify the script to parse the filename from the full URL. This was easy with Python’s urllib. Now we are in the process of implementing the presentation API of IIIF. During the Wikimania Hackathon we also added a link to IIIF to the user interface of Ajapaik.

-Siru, intern