Kirjoittajan arkistot:siru

Upscaling Images in Helsinki Rephotography

It is the beginning of a new month, so it’s time again for me to write a summary about what we have been doing at Helsinki Rephotography in September. Last month was the Hack4OpenGLAM event, so most of our time went to preparing for that. We were participating with a project about upscaling images with machine learning and artificial neural networks. For this we used the OpenCV library which stands for Open Computer Vision.

Artificial neural networks are computing systems that are inspired by the real biological neural networks found in our brains. They can be used for example in pattern recognition, 3D reconstruction and machine translation. For testing, we used a neural network model trained for upscaling images called EDSR.

Setting everything up

I was the one responsible for doing the initial legwork and getting OpenCV up and running. This required compiling it from the source code, since the super resolution module necessary for upscaling images was not included in the pre-compiled version of OpenCV. I first did a test version without GPU acceleration just to see how fast the upscaling was with just the CPU power. This ended up working with simpler upscaling models such as FSRCNN, but that one obviously wasn’t as good as EDSR, which took well over a minute to upscale a small image with just the CPU.

I then figured out how to get CUDA working with OpenCV. This allowed the use of Nvidia GPUs in the upscaling process. It required installing the CUDA toolkit and recompiling OpenCV with support for it. After setting it up, I ran into another problem: my GPU had too little integrated memory to do the upscaling. This prompted Kimmo to set up an analytics server with a more powerful GPU that we could try, and this ended up working. It now only took 5 seconds to upscale the same picture that took over a minute just with the CPU.

The results

After this we could try it on different images and see how it worked. We ended up using 3 different image upscaling models; GFPGAN, Real-ESRGAN and ESRGAN. These models were based on a different neural network technique called generative adversarial network. In this technique, two neural networks compete against each other. The other creates data points, while the other tries to distinguish between the real ones and the ones created by the neural network.

The end result was that the images themselves looked good, but the AI sometimes tended to create weird things, such as badly shaven mustaches, faces inside faces and walls that were made from fabric. So, it was not perfect, but could work if you weren’t looking for the perfect result. Here is a good example of the upscaling that we did on an image provided to us by Fortepan, who we did this project in collaboration with. This image was upscaled with ESRGAN. In it you can clearly see the difference in sharpness and details between the original and the upscaled image.

For me it was interesting to see and learn what machine learning and neural networks can be used for. I hope that in the future we can also use this for something useful.

-Siru, intern

August in Helsinki Rephotography

This month in Helsinki Rephotography we have worked on implementing IIIF using IIIPImage to Ajapaik. IIIF stands for the International Image Interoperability Framework, and it is a protocol for standardizing image retrieval. Using IIIF it is easy for users to manipulate images, like modifying size, scale, region of interest, rotation, quality, and format. IIIF also allows for easy implementation of image annotations, and with it users can comment, transcribe and draw on images.

The IIIF protocol has two parts, the image API, and the presentation API. The image API delivers images in a way that easily specifies the characteristics of the image, such as region, size, rotation, and quality. The presentation API on the other hand provides the information necessary to drive a compelling viewer experience.

IIPImage is a high-performance image server that can be used to stream high resolution images. It uses IIIF and the Fast CGI protocol in combination with a front-end web server (nginx in Ajapaik’s case). IIPImage can use images that are either in the TIFF format, or the JPEG2000 format. It also comes with an image converting tool called vips than can be used to convert images to the TIFF format.

During the month I have made a Python script that is used with the uWSGI server to download images from Wikimedia, convert them to the TIFF format and then display them over IIIF. uWSGI is a Web Server Gateway Interface server implementation that can be used in conjunction with a web server to forward requests to an external program.

The idea is to get this script to execute whenever a 404 Not Found error happens at the IIPImage server. It will parse the filename from the request and find out if the image can be found on Wikimedia Commons. If it can, it will download and convert the image to TIFF using vips. Lastly it will redirect the user to the IIPImage server that will serve the image using the IIIF protocol.

I did this by launching uWSGI with the Python script and a socket file as parameters. I then added a new location to the nginx site configuration that specifies that the requests to this location should be forwarded to the uWSGI server. This worked when the user directly accessed the location but didn’t when they were forwarded to it from a 404 error, since in that case only the request URL got forwarded to uWSGI instead of the full parameters.

To fix this we had to modify the script to parse the filename from the full URL. This was easy with Python’s urllib. Now we are in the process of implementing the presentation API of IIIF. During the Wikimania Hackathon we also added a link to IIIF to the user interface of Ajapaik.

-Siru, intern

Helsinki Rephotography From The Eyes of an Intern

My name is Siru, and I am an IT student at the Vaasa University of Applied Sciences. Currently I am an intern at Wikimedia Finland with the Helsinki Rephotography project, which is a collaboration project between Wikimedia Finland and the Estonian Ajapaik. The idea is to create a mobile app that allows users to rephotograph old photographs on Ajapaik with new, more modern versions and to see how those places have changed. 

Currently we are helping Ajapaik with their server migration, since they are in the process of upgrading to a more powerful server. We have now switched to using the new server and I have setup an automated backup system and a downtime notification system that sends notifications to Slack in case of errors. 

The backup system is a bash script that is made to run daily on both Ajapaik and the backup server. On Ajapaik the script makes a backup of the database and then compresses, encrypts, and digitally signs it. It also makes an unencrypted backup that only contains specific tables from the database. 

On the backup server the script first checks the available disk space and sends a notification in case it is running low. It then syncs the pictures and database backups from Ajapaik to a backup folder. After syncing it verifies the integrity of the database backups using the digital signature. If the syncing fails for some reason and/or the integrity can’t be verified, it then sends a notification. 

I have also made several other bash scripts, such as one that monitors the resources of Ajapaik and sends a notification in case disk space is running low, if CPU load is high for extended periods or if the memory or swap memory are running low. There is also a script that runs on startup of the Ajapaik server. It basically starts all the services and checks that Ajapaik is reachable with HTTP. There is also a similar script on the backup server that constantly monitors if the Ajapaik website is reachable and in case it isn’t it sends a notification. 

In the future I will be participating in the programming of the mobile app which we are programming in Flutter. It has been interesting to do the system maintenance tasks since that is something I am interested in and that I already have experience in from working with my own servers. The internship is being done remotely, which has its own challenges, but I think it works perfectly for this kind of job. 

-Siru, intern