Try to using TecoGAN(Temporally Coherent GANs) super-resolution conversion to change a image from low to high resolution
i tried to use this TecoGAN posted on github.
Input low-res image 180*144 (png file)
Output result high-res image 720*576 (png file)
Converted result is the image to 4 times the resolution of the vertical and horizontal.
Let’s compare the converted image with the 4 times enlarged image (Gimp) and the Ground Truth image. (Slider can be moved left and right).
Left: 4x enlarged image(Lanczos3) <-> Right: Super-res image (TecoGAN)
Left: Ground Truth image <-> Right: Super-res image (TecoGAN)
Left: 4x(Lanczos3) <-> Right: 4x(TecoGAN)
Original movie: 512×512 -> 4x enlarge 2024×2024
How to use (Usage)
- Linux Ubuntu 18.04 LTS (Windows10 (1903) + WSL)
Prepare for python environment
i used python3.7.
Activate python virtual environment.
virtualenv -p python3.7 venv37 source venv37/bin/activate
Install required libraries
If does not specify the tensorflow version, 2.x will be installed, so let specify 1.x.
$ pip install tensorflow-gpu==1.15
Also install ‘unzip’
It will be used to decompress the downloaded data, so install ‘unzip’.
$ sudo apt install unzip
Prepare for TecoGAN
Clone TecoGAN from github repository.
$ git clone https://github.com/thunil/TecoGAN.git $ cd TecoGAN
Install the necessary libraries for TecoGAN with the following.
$ pip install -r requirements.txt
Check what was installed (excerpt)
$ pip freeze Keras==2.3.1 matplotlib==3.1.2 numpy==1.17.4 opencv-python==184.108.40.206 pandas==0.25.3 scikit-image==0.16.2 scipy==1.3.3 tensorflow-gpu==1.15.0 torch==1.3.1 torchvision==0.4.2
Prepare for traind model and test data
Run the following to download the traind model, test data.
$ python runGan.py 0
When download is complete, it will be in the following state (it took several tens of minutes).
TecoGAN/ HR/ bridge/ calendar/ city/ face/ foliage/ room/ walk/ LR/ bridge/ calendar/ city/ face/ foliage/ room/ walk/ model/ TecoGAN.data-00000-of-00001 TecoGAN.index TecoGAN.meta
Run inference (conversion from low to high resolution).
$ python runGan.py 1
It takes about 30 seconds.
$ python runGan.py 1 Testing test case 1 WARNING:tensorflow: ...omit... Loading weights from ckpt model Frame evaluation starts!! Warming up 5 Warming up 4 Warming up 3 Warming up 2 Warming up 1 saving image output_0001 saving image output_0002 ...omit... saving image output_0041 total time 35.60395073890686, frame number 46 $
When inference is completed, the super-resolution converted image is output under results as shown below.
TecoGAN/ results/ calendar/ output_0001.png ... output_0041.png log/ logfile.txt
How to try super-res conversion on other images?
At line 70 of TecoGAN/runGan.py, ‘TecoGAN/LR/calendar’ is specified as the input image directory (–input_dir_LR).
So, modify this to any directory, place other image file(.png) in it, and run inference.
elif( runcase == 1 ): # inference a trained model dirstr = './results/' # the place to save the results testpre = ['calendar'] # the test cases ***modify this to any directory
At first, i tried as a still image conversion, but if you want to convert a movie, you can extract all the frames as a still image with ‘ffmpeg’, run super resolution conversion, and connect with ffmpeg to return to the movie.
***Frame interpolation of movie in the sense of super resolution in the time-axis