Try to using TecoGAN(Temporally Coherent GANs) super-resolution conversion to change a image from low to high resolution

Index

Tried it
1. ToDo
How to use (Usage)

Tried it

ToDo

i tried to use this TecoGAN posted on github.

GitHub - thunil/TecoGAN: This repo contains source code and materials for the TEmporally COherent GAN SIGGRAPH project.

This repo contains source code and materials for the TEmporally COherent GAN SIGGRAPH project. - thunil/TecoGAN

Paper: Learning Temporal Coherence via Self-Supervision for GAN-based Video Generation

Input low-res image 180*144 (png file)

Output result high-res image 720*576 (png file)

Done!

Converted result is the image to 4 times the resolution of the vertical and horizontal.

Let’s compare the converted image with the 4 times enlarged image (Gimp) and the Ground Truth image. (Slider can be moved left and right).

Left: 4x enlarged image(Lanczos3) <-> Right: Super-res image (TecoGAN)

Left: Ground Truth image <-> Right: Super-res image (TecoGAN)

Left: 4x(Lanczos3) <-> Right: 4x(TecoGAN)

Original movie: 512×512 -> 4x enlarge 2024×2024

How to use (Usage)

Execution environment

Linux Ubuntu 18.04 LTS (Windows10 (1903) + WSL)

It seems that GPU is not used when running on WSL…

With using GPU, how to run python of Anaconda on WSL (operation confirmation with NNabla)

It seems that GPU (CUDA + cuDNN) works by running python of Anaconda environment on WSL (Windows Subsystem for Linux), so try it

Prepare for python environment

i used python3.7.

Activate python virtual environment.

$ virtualenv -p python3.7 venv37
$ source venv37/bin/activate
(venv37)$

Install required libraries

If does not specify the tensorflow version, 2.x will be installed, so let specify 1.x.

$ pip install tensorflow-gpu==1.15

Also install ‘unzip’

It will be used to decompress the downloaded data, so install ‘unzip’.

$ sudo apt install unzip

Prepare for TecoGAN

Clone TecoGAN from github repository.

$ git clone https://github.com/thunil/TecoGAN.git
$ cd TecoGAN

Install the necessary libraries for TecoGAN with the following.

$ pip install -r requirements.txt

Check what was installed (excerpt)

$ pip freeze
Keras==2.3.1
matplotlib==3.1.2
numpy==1.17.4
opencv-python==4.1.2.30
pandas==0.25.3
scikit-image==0.16.2
scipy==1.3.3
tensorflow-gpu==1.15.0
torch==1.3.1
torchvision==0.4.2

Prepare for traind model and test data

Run the following to download the traind model, test data.

$ python runGan.py 0

When download is complete, it will be in the following state (it took several tens of minutes).

TecoGAN/
  HR/
    bridge/
    calendar/
    city/
    face/
    foliage/
    room/
    walk/
  LR/
    bridge/
    calendar/
    city/
    face/
    foliage/
    room/
    walk/
  model/
    TecoGAN.data-00000-of-00001
    TecoGAN.index
    TecoGAN.meta

Run

Run inference (conversion from low to high resolution).

$ python runGan.py 1

It takes about 30 seconds.

$ python runGan.py 1
Testing test case 1
WARNING:tensorflow:
...omit...
Loading weights from ckpt model
Frame evaluation starts!!
Warming up 5
Warming up 4
Warming up 3
Warming up 2
Warming up 1
saving image output_0001
saving image output_0002
...omit...
saving image output_0041
total time 35.60395073890686, frame number 46
$

When inference is completed, the super-resolution converted image is output under results as shown below.

TecoGAN/
  results/
    calendar/
      output_0001.png
      ...
      output_0041.png
  log/
    logfile.txt

How to try super-res conversion on other images?

At line 70 of TecoGAN/runGan.py, ‘TecoGAN/LR/calendar’ is specified as the input image directory (–input_dir_LR).

So, modify this to any directory, place other image file(.png) in it, and run inference.

elif( runcase == 1 ): # inference a trained model

  dirstr = './results/' # the place to save the results
  testpre = ['calendar'] # the test cases  ***modify this to any directory

At first, i tried as a still image conversion, but if you want to convert a movie, you can extract all the frames as a still image with ‘ffmpeg’, run super resolution conversion, and connect with ffmpeg to return to the movie.