Deep Learning Algorithms in Super-Resolution Images
Subject Areas : Communication Engineering
Bahar Ghaderi
1
(
Department of Electrical Engineering, Shiraz Branch, Islamic Azad University, Shiraz, Iran
)
Hamid Azad
2
(
Department of Electrical Engineering, Shiraz Branch, Islamic Azad University, Shiraz, Iran
)
Keywords: Super-resolution (SR), Deep Learning Algorithms, High Resolution (HR) Image, Low Resolution (LR) Image, Convolutional Neural Network,
Abstract :
Image super-resolution is one of the important image processing processes to increase the resolution of images and videos. In recent years, methods based on deep neural networks for super-resolution have seen significant progress. The aim of this paper is to provide a comprehensive review on recent developments in super-resolution image using deep learning approaches. In this article, while introducing the concepts of image super-resolution, the common deep learning algorithms for super-resolution and the applications of super-resolution have been investigated. In addition, the set of databases and evaluation criteria are described. This article can open the way for image processing researchers in the super-resolution process. The authors’ effort has been to explore all aspects of this process.
[1] F. Liu, X. Yang, and B. De Baets, "A deep recursive multi-scale feature fusion network for image super-resolution," Journal of Visual Communication and Image Representation, vol. 90, p. 103730, 2023.
[2] L. Inzerillo, F. Acuto, G. Di Mino, and M. Z. Uddin, "Super-resolution images methodology applied to UAV datasets to road pavement monitoring," Drones, vol. 6, p. 171, 2022.
[3] Y. Huang, L. Shao, and A. F. Frangi, "Simultaneous super-resolution and cross-modality synthesis of 3D medical images using weakly-supervised joint convolutional sparse coding," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 6070-6079.
[4] D. Yang, Z. Li, Y. Xia, and Z. Chen, "Remote sensing image super-resolution: Challenges and approaches," in 2015 IEEE international conference on digital signal processing (DSP), 2015, pp. 196-200.
[5] J. Jiang, C. Wang, X. Liu, and J. Ma, "Deep learning-based face super-resolution: A survey," ACM Computing Surveys (CSUR), vol. 55, pp. 1-36, 2021.
[6] A. B. Deshmukh and N. Usha Rani, "Fractional-Grey Wolf optimizer-based kernel weighted regression model for multi-view face video super resolution," International Journal of Machine Learning and Cybernetics, vol. 10, pp. 859-877, 2019.
[7] H. Liu, Z. Ruan, P. Zhao, C. Dong, F. Shang, Y. Liu, et al., "Video super-resolution based on deep learning: a comprehensive survey," Artificial Intelligence Review, vol. 55, pp. 5981-6035, 2022.
[8] S. Anwar, S. Khan, and N. Barnes, "A deep journey into super-resolution: A survey," ACM Computing Surveys (CSUR), vol. 53, pp. 1-34, 2020.
[9] C. Qiao, D. Li, Y. Liu, S. Zhang, K. Liu, C. Liu, et al., "Rationalized deep learning super-resolution microscopy for sustained live imaging of rapid subcellular processes," Nature biotechnology, vol. 41, pp. 367-377, 2023.
[10] M. Chaika, S. Afat, D. Wessling, C. Afat, D. Nickel, S. Kannengiesser, et al., "Deep learning-based super-resolution gradient echo imaging of the pancreas: Improvement of image quality and reduction of acquisition time," Diagnostic and Interventional Imaging, vol. 104, pp. 53-59, 2023.
[11] B. Niu, Q. Feng, J. Yang, B. Chen, B. Gao, J. Liu, et al., "Solid waste mapping based on very high resolution remote sensing imagery and a novel deep learning approach," Geocarto International, vol. 38, p. 2164361, 2023.
[12] G. Liang, U. KinTak, H. Yin, J. Liu, and H. Luo, "Multi-scale hybrid attention graph convolution neural network for remote sensing images super-resolution," Signal Processing, vol. 207, p. 108954, 2023.
[13] W. Yang, X. Zhang, Y. Tian, W. Wang, J.-H. Xue, and Q. Liao, "Deep learning for single image super-resolution: A brief review," IEEE Transactions on Multimedia, vol. 21, pp. 3106-3121, 2019.
[14] Y. Li, B. Sixou, and F. Peyrin, "A review of the deep learning methods for medical images super resolution problems," Irbm, vol. 42, pp. 120-133, 2021.
[15] Y. LeCun, "LeNet-5, convolutional neural networks," URL: http://yann. lecun. com/exdb/lenet, vol. 20, p. 14, 2015.
[16] Y. Luo, L. Zhou, S. Wang, and Z. Wang, "Video satellite imagery super resolution via convolutional neural networks," IEEE Geoscience and Remote Sensing Letters, vol. 14, pp. 2398-2402, 2017.
[17] K. Umehara, J. Ota, and T. Ishida, "Application of super-resolution convolutional neural network for enhancing image resolution in chest CT," Journal of digital imaging, vol. 31, pp. 441-450, 2018.
[18] M. U. Müller, N. Ekhtiari, R. M. Almeida, and C. Rieke, "Super-resolution of multispectral satellite images using convolutional neural networks," arXiv preprint arXiv:2002.00580, 2020.
[19] M. Taş and B. Yılmaz, "Super resolution convolutional neural network based pre-processing for automatic polyp detection in colonoscopy images," Computers & Electrical Engineering, vol. 90, p. 106959, 2021.
[20] H. M. Keshk and X.-C. Yin, "Satellite super-resolution images depending on deep learning methods: a comparative study," in 2017 IEEE International Conference on Signal Processing, Communications and Computing (ICSPCC), 2017, pp. 1-7.
[21] C. Tian, R. Zhuge, Z. Wu, Y. Xu, W. Zuo, C. Chen, et al., "Lightweight image super-resolution with enhanced CNN," Knowledge-Based Systems, vol. 205, p. 106235, 2020.
[22] J. Yamanaka, S. Kuwashima, and T. Kurita, "Fast and accurate image super resolution by deep CNN with skip connection and network in network," in Neural Information Processing: 24th International Conference, ICONIP 2017, Guangzhou, China, November 14-18, 2017, Proceedings, Part II 24, 2017, pp. 217-225.
[23] C. Dong, C. C. Loy, K. He, and X. Tang, "Learning a deep convolutional network for image super-resolution," in European conference on computer vision, 2014, pp. 184-199.
[24] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, et al., "Generative adversarial nets," Advances in neural information processing systems, vol. 27, 2014.
[25] J. Zhu, C. Tan, J. Yang, G. Yang, and P. Lio’, "Arbitrary Scale Super-Resolution for Medical Images," International journal of neural systems, vol. 31, p. 2150037, 2021.
[26] Y. Gu, Z. Zeng, H. Chen, J. Wei, Y. Zhang, B. Chen, et al., "MedSRGAN: medical images super-resolution using generative adversarial networks," Multimedia Tools and Applications, vol. 79, pp. 21815-21840, 2020.
[27] H. Liu, J. Liu, S. Hou, T. Tao, and J. Han, "Perception consistency ultrasound image super-resolution via self-supervised CycleGAN," Neural Computing and Applications, pp. 1-11, 2021.
[28] D. Mahapatra, B. Bozorgtabar, and R. Garnavi, "Image super-resolution using progressive generative adversarial networks for medical image analysis," Computerized Medical Imaging and Graphics, vol. 71, pp. 30-39, 2019.
[29] X. Bing, W. Zhang, L. Zheng, and Y. Zhang, "Medical image super resolution using improved generative adversarial networks," IEEE Access, vol. 7, pp. 145030-145038, 2019.
[30] X. Yu and F. Porikli, "Ultra-resolving face images by discriminative generative networks," in European conference on computer vision, 2016, pp. 318-333.
[31] L. R. Medsker and L. Jain, "Recurrent neural networks," Design and Applications, vol. 5, p. 2, 2001.
[32] Y. Liu, D. Yang, F. Zhang, Q. Xie, and C. Zhang, "Deep recurrent residual channel attention network for single image super-resolution," The Visual Computer, pp. 1-16, 2023.
[33] W. Weng, Y. Zhang, and Z. Xiong, "Boosting event stream super-resolution with a recurrent neural network," in European Conference on Computer Vision, 2022, pp. 470-488.
[34] W. Han, S. Chang, D. Liu, M. Yu, M. Witbrock, and T. S. Huang, "Image super-resolution via dual-state recurrent networks," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 1654-1663.
[35] Y. Fu, Z. Liang, and S. You, "Bidirectional 3D quasi-recurrent neural network for hyperspectral image super-resolution," IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 14, pp. 2674-2688, 2021.
[36] M. R. Arefin, V. Michalski, P.-L. St-Charles, A. Kalaitzis, S. Kim, S. E. Kahou, et al., "Multi-image super-resolution for remote sensing using deep recurrent networks," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020, pp. 206-207.
[37] J. Kim, J. K. Lee, and K. M. Lee, "Deeply-recursive convolutional network for image super-resolution," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 1637-1645.
[38] W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang, "Fast and accurate image super-resolution with deep laplacian pyramid networks," IEEE transactions on pattern analysis and machine intelligence, vol. 41, pp. 2599-2613, 2018.
[39] S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural computation, vol. 9, pp. 1735-1780, 1997.
[40] K. Greff, R. K. Srivastava, J. Koutník, B. R. Steunebrink, and J. Schmidhuber, "LSTM: A search space odyssey," IEEE transactions on neural networks and learning systems, vol. 28, pp. 2222-2232, 2016.
[41] Y. Chang and B. Luo, "Bidirectional convolutional LSTM neural network for remote sensing image super-resolution," Remote Sensing, vol. 11, p. 2333, 2019.
[42] H. Zhu, M. Guo, H. Li, Q. Wang, and A. Robles-Kelly, "Breaking the spatio-angular trade-off for light field super-resolution via lstm modelling on epipolar plane images," arXiv preprint arXiv:1902.05672, 2019.
[43] C. Chou, J. Park, and E. Chou, "Generating high-resolution climate change projections using super-resolution convolutional LSTM neural networks," in 2021 13th International Conference on Advanced Computational Intelligence (ICACI), 2021, pp. 293-298.
[44] X. Lu, X. Liu, L. Zhang, F. Jia, and Y. Yang, "Hyperspectral image super-resolution based on attention ConvBiLSTM network," International Journal of Remote Sensing, vol. 43, pp. 5059-5074, 2022.
[45] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778.
[46] T. Lu, J. Wang, Y. Zhang, Z. Wang, and J. Jiang, "Satellite image super-resolution via multi-scale residual deep neural network," Remote Sensing, vol. 11, p. 1588, 2019.
[47] W. Muhammad, Z. Bhutto, S. Masroor, M. H. Shaikh, J. Shah, and A. Hussain, "IRMIRS: Inception-ResNet-Based Network for MRI Image Super-Resolution," CMES-Computer Modeling in Engineering & Sciences, vol. 136, 2023.
[48] Q. Qin, J. Dou, and Z. Tu, "Deep ResNet based remote sensing image super-resolution reconstruction in discrete wavelet domain," Pattern Recognition and Image Analysis, vol. 30, pp. 541-550, 2020.
[49] D.-W. Jang and R.-H. Park, "Densenet with deep residual channel-attention blocks for single image super resolution," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2019, pp. 0-0.
[50] S. Bell-Kligler, A. Shocher, and M. Irani, "Blind super-resolution kernel estimation using an internal-gan," Advances in Neural Information Processing Systems, vol. 32, 2019.
[51] E. Agustsson and R. Timofte, "Ntire 2017 challenge on single image super-resolution: Dataset and study," in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2017, pp. 126-135.
[52] J. Cai, H. Zeng, H. Yong, Z. Cao, and L. Zhang, "Toward real-world single image super-resolution: A new benchmark and a new model," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 3086-3095.
[53] P. Wei, Z. Xie, H. Lu, Z. Zhan, Q. Ye, W. Zuo, et al., "Component divide-and-conquer for real-world image super-resolution," in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VIII 16, 2020, pp. 101-117.
[54] C. Chen, Z. Xiong, X. Tian, Z.-J. Zha, and F. Wu, "Camera lens super-resolution," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 1652-1660.
[55] X. Zhang, Q. Chen, R. Ng, and V. Koltun, "Zoom to learn, learn to zoom," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 3762-3770.
[56] W. Wang, E. Xie, X. Liu, W. Wang, D. Liang, C. Shen, et al., "Scene text image super-resolution in the wild," in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part X 16, 2020, pp. 650-666.
[57] T. Köhler, M. Bätz, F. Naderi, A. Kaup, A. Maier, and C. Riess, "Toward bridging the simulated-to-real gap: Benchmarking super-resolution on real data," IEEE transactions on pattern analysis and machine intelligence, vol. 42, pp. 2944-2959, 2019.
[58] H. R. V. Joze, I. Zharkov, K. Powell, C. Ringler, L. Liang, A. Roulston, et al., "Imagepairs: Realistic super resolution dataset via beam splitter camera rig," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020, pp. 518-519.
[59] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image quality assessment: from error visibility to structural similarity," IEEE transactions on image processing, vol. 13, pp. 600-612, 2004.
[60] H. R. Sheikh, A. C. Bovik, and G. De Veciana, "An information fidelity criterion for image quality assessment using natural scene statistics," IEEE Transactions on image processing, vol. 14, pp. 2117-2128, 2005.
[61] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, "The unreasonable effectiveness of deep features as a perceptual metric," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 586-595.
[62] A. Mittal, R. Soundararajan, and A. C. Bovik, "Making a “completely blind” image quality analyzer," IEEE Signal processing letters, vol. 20, pp. 209-212, 2012.
[63] C. Ma, C.-Y. Yang, X. Yang, and M.-H. Yang, "Learning a no-reference quality metric for single-image super-resolution," Computer Vision and Image Understanding, vol. 158, pp. 1-16, 2017.
[64] N. Venkatanath, D. Praneeth, M. C. Bh, S. S. Channappayya, and S. S. Medasani, "Blind image quality evaluation using perception based features," in 2015 twenty first national conference on communications (NCC), 2015, pp. 1-6.
[65] L. Hitachi, "Super-resolution technology to convert video of various resolutions to high-definition," ed.
[66] Y. Wang, R. Fevig, and R. R. Schultz, "Super-resolution mosaicking of UAV surveillance video," in 2008 15th IEEE International Conference on Image Processing, 2008, pp. 345-348.
[67] M. D. Robinson, S. J. Chiu, C. A. Toth, J. A. Izatt, J. Y. Lo, and S. Farsiu, "New applications of super-resolution in medical imaging," in Super-resolution imaging, ed: CRC Press, 2017, pp. 383-412.
[68] R. Willett, I. Jermyn, R. Nowak, and J. Zerubia, "Wavelet-based superresolution in astronomy," ed: Astronomical Society of the Pacific, 2003.
[69] W. Liu, D. Lin, and X. Tang, "Hallucinating faces: Tensorpatch super-resolution and coupled residue compensation," in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), 2005, pp. 478-484.
[70] Z. Yuan, J. Wu, S.-i. Kamata, A. Ahrary, and P. Yan, "Fingerprint image enhancement by super resolution with early stopping," in 2009 IEEE International Conference on Intelligent Computing and Intelligent Systems, 2009, pp. 527-531.
[71] G. Fahmy, "Super-resolution construction of iris images from a visual low resolution face video," in 2007 9th International Symposium on Signal Processing and Its Applications, 2007, pp. 1-4.
