Standalone Test: SD->HD & HD->4K Video Upscaling
While the NTIRE2017 competition forms are very useful benchmark for the state of affairs in image upscaling, the highly-textured image content used in the competition may not fully represent the typical conditions found in real-world video sequences.
Therefore, in a second test, we downscaled sixteen 4K and twenty-two HD uncompressed video sequences from xiph.org (using Matlab’s imresize() function) in order to assess upscaling performance on standard video sequences. The results of Table 2 show the performance of bicubic and Lanczos filter (under OpenCV’s implementation, similar results were obtained with various Lanczos kernels, edge-adaptive filters and other polyphase interpolators), as well as:
- the SNU_CVLab2 solution that was the winning solution of the NTIRE2017 competition;
- the well-known FSRCNN method from CVPR 2016 .
The iSize upscaler offers more than 1.2dB average improvement against linear filters, while achieving runtime per frame that approaches that of bicubic or Lanczos upscaling. Across the different sequences, improvement ranges from 0.6dB to 2.2dB, depending on the content of each video sequence. Our solution outperforms the NTIRE2017 winning solution. The closest competitor to our approach is FSRCNN : for the SDàHD upscaling, FSRCNN offers 0.32dB higher PSNR to our proposal, albeit at more than 130-fold increase in execution time per frame. Importantly, FSRCNN is outperformed by the iSize upscaler for the HDà4K upscaling tests.
Table 2. Summary of PSNR, percentile reduction of mean squared error against bicubic upscaling, and execution time per frame for: linear upscalers, the NTIRE2017 competition winner and iSize. Unless noted otherwise, all methods were executed on GPU. Execution times include both GPU time and data transfers. The system spec was: Intel Core i5-3570K, 16 GB RAM, NVidia GTX 1070.
Given that the video sequences used for the results of Table 2 can contain many frames with limited texture content, the average PSNR difference shown by iSize and FSRCNN against linear upscalers is not as high as for the case of the highly-textured NTIRE2017 images. However, beyond the standard upscaling that is optimized for PSNR, iSize’s unique IP for upscaler “boosting” leads to significant quality improvement, as shown by the visual examples of the next page. This further visual quality improvement incurs 35% overhead (on average) in comparison to the iSize timing results reported in Table 1 and Table 2. The iSize upscaler boosting can be tuned adaptively (from none to some), thereby allowing for tuning of our solution according to the type of content, upscaled resolution and display settings used by the client device.
 The “Bicubic” (classic) track is meant to facilitate the easy deployment of recent proposed methods for the task for example-based single-image super-resolution. It assumes that the degradation operators are the same as commonly used in the recent super-resolution literature. For obtaining the low res images, the Matlab function “imresize” is used, with default settings (bicubic interpolation) and downscaling factor 2. This was tested on a large newly collected dataset -DIV2K- of images with a large diversity of contents. More information can be found on the competition page .
 CPU: https://ark.intel.com/products/65520/Intel-Core-i5-3570K-Processor-6M-Cache-up-to-3_80-GHz
- Timofte, et al., “NTIRE 2017 challenge on single image super-resolution: Methods and results,” Proc. Comp. Vis. and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conf. on Comp. Vis. and Pattern Recognition, CVPR, IEEE, 2017, https://goo.gl/TQRT7E.
- Lim, et al. “Enhanced deep residual networks for single image super-resolution,” Proc. Comp. Vis. and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conf. on Comp. Vis. and Pattern Recogn., CVPR, IEEE, 2017, https://goo.gl/PDSTiV.
- Dong, et al., “Accelerating the super-resolution convolutional neural network,” Proc. 2016 IEEE Conf. on Comp. Vis. and Pattern Recognition, CVPR, IEEE, 2016, https://goo.gl/Qa1UmX.