Skip to main content
SpringerLink
Log in

Single Image Super-Resolution Based on Capsule Neural Networks

  • Conference paper
  • First Online:
Intelligent Systems (BRACIS 2023)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14197))

Included in the following conference series:

  • 228 Accesses

Abstract

Single image super-resolution (SISR) consists of obtaining one high-resolution version of a low-resolution image by increasing the number of pixels per unit area. This method has been actively investigated by the research community, due to the wide variety of problems ranging from real-world surveillance to aerial and satellite imaging. Most of the improvements in SISR come from convolutional networks, in which approaches often focus on the deeper and wider architectural paradigm. In this work, we decided to step up from the traditional convolutions and adopt the concept of capsules. Since their overwhelming results in image classification and segmentation problems, we question how suitable they are for SISR. We also verify that different solutions share similar configurations, and argue that this trend leads to fewer explorations of network designs. Throughout our experiments, we check various strategies to improve results, ranging from new and different loss functions to changes in the capsule layers. Our network achieved positive and promising results with fewer convolutional-based layers, showing that capsules might be a concept worth applying to the image super-resolution problem. In particular, we observe that the proposed method recreates the connection between the different characters more precisely, thus demonstrating the potential of capsules in super-resolution problems.

Artur Jordáo: This work was done when Artur Jordao was a post-doctoral researcher at the University of Campinas.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    github.com/S-aiueo32/sr-pytorch-lightning.

  2. 2.

    https://github.com/jettify/pytorch-optimizer.

  3. 3.

    https://www.comet.ml/.

  4. 4.

    https://docs.ray.io/en/master/tune/index.html.

References

  1. Abadi, M., et al.: TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems (2015). Software available from tensorflow.org

    Google Scholar 

  2. Agustsson, E., Timofte, R.: NTIRE 2017 challenge on single image super-resolution: dataset and study. In: CVPR Workshops, pp. 1–8 (2017)

    Google Scholar 

  3. Andersson, P., Nilsson, J., Akenine-Möller, T., Oskarsson, M., Åström, K., Fairchild, M.D.: FLIP: a difference evaluator for alternating images. In: ACM on Computer Graphics and Interactive Techniques (2020)

    Google Scholar 

  4. Anwar, S., Khan, S.H., Barnes, N.: A deep journey into super-resolution: a survey. ACM Comput. Surv. 53(3), 60:1–60:34 (2020). https://doi.org/10.1145/3390462

  5. Barron, J.T.: A More General Robust Loss Function. arXiv preprint arXiv:1701.03077 (2017)

  6. Behjati, P., Rodriguez, P., Mehri, A., Hupont, I., Tena, C.F., Gonzalez, J.: OverNet: lightweight multi-scale super-resolution with overscaling network. In: WACV, pp. 1–11 (2021)

    Google Scholar 

  7. Bevilacqua, M., Roumy, A., Guillemot, C., Alberi-Morel, M.L.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: BMVC (2012)

    Google Scholar 

  8. Cai, J., Gu, S., Timofte, R., Zhang, L.: NTIRE 2019 challenge on real image super-resolution: methods and results. In: CVPR Workshops, pp. 1–8 (2019)

    Google Scholar 

  9. Canny, J.: A computational approach to edge detection. Trans. Pattern Anal. Mach. Intell. 8(6), 679–698 (1986)

    Article  Google Scholar 

  10. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)

    Article  Google Scholar 

  11. Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 184–199. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10593-2_13

    Chapter  Google Scholar 

  12. Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 391–407. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_25

    Chapter  Google Scholar 

  13. Gu, J., Tresp, V.: Improving the robustness of capsule networks to image affine transformations. In: CVPR, pp. 1–15 (2020)

    Google Scholar 

  14. Gu, J., Wu, B., Tresp, V.: Effective and efficient vote atack on capsule networks. In: ICLR (2021)

    Google Scholar 

  15. Hinton, G., Sabour, S., Frosst, N.: Matrix capsules with EM routing. In: ICLR, pp. 1–10 (2018)

    Google Scholar 

  16. Hinton, G.E., Krizhevsky, A., Wang, S.D.: Transforming auto-encoders. In: Artificial Neural Networks and Machine Learning, pp. 44–51 (2011)

    Google Scholar 

  17. Hore, A., Ziou, D.: Image quality metrics: PSNR vs. SSIM. In: ICPR, pp. 2366–2369 (2010)

    Google Scholar 

  18. Hsu, J., Kuo, C., Chen, D.: Image super-resolution using capsule neural networks. IEEE Access (2020)

    Google Scholar 

  19. Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: CVPR, pp. 1–9 (2015)

    Google Scholar 

  20. Huang, W., Zhou, F.: DA-CapsNet: dual attention mechanism capsule network. Sci. Rep. (2020)

    Google Scholar 

  21. Irani, M., Peleg, S.: Improving resolution by image registration. In: CVGIP: Graph. Model. Image Process. 53(3), 231–239 (1991)

    Google Scholar 

  22. Ji, X., Cao, Y., Tai, Y., Wang, C., Li, J., Huang, F.: Real-world super-resolution via Kernel estimation and noise injection. In: CVPR Workshops, pp. 1–8 (2020)

    Google Scholar 

  23. Kastryulin, S., Zakirov, D., Prokopenko, D.: PyTorch image quality: metrics and measure for image quality assessment (2019). https://github.com/photosynthesis-team/piq

  24. Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: CVPR, pp. 1–8 (2016)

    Google Scholar 

  25. Kim, J., Kwon Lee, J., Mu Lee, K.: Deeply-recursive convolutional network for image super-resolution. In: CVPR, pp. 1–13 (2016)

    Google Scholar 

  26. Kim, J.H., Lee, J.S.: Deep residual network with enhanced upscaling module for super-resolution. In: CVPR Workshops, pp. 1–15 (2018)

    Google Scholar 

  27. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Bengio, Y., LeCun, Y. (eds.) ICLR (2015)

    Google Scholar 

  28. LaLonde, R., Bagci, U.: Capsules for Object Segmentation. arXiv preprint arXiv:1804.04241 (2018)

  29. Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    Article  Google Scholar 

  30. Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: CVPR, pp. 1–8 (2017)

    Google Scholar 

  31. Li, C., Bovik, A.C.: Content-weighted video quality assessment using a three-component image model. J. Electron. Imag. 19, 19 (2010)

    Google Scholar 

  32. Liaw, R., Liang, E., Nishihara, R., Moritz, P., Gonzalez, J.E., Stoica, I.: Tune: a research platform for distributed model selection and training. arXiv preprint arXiv:1807.05118 (2018)

  33. Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. In: CVPR Workshops, pp. 1–8 (2017)

    Google Scholar 

  34. Lin, M., Chen, Q., Yan, S.: Network in Network. arXiv preprint arXiv:1312.4400 (2013)

  35. Majdabadi, M.M., Ko, S.B.: Capsule GAN for Robust Face Super-Resolution. Multim. Tools Appl. (2020)

    Google Scholar 

  36. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: ICCV, vol. 2, pp. 416–423 (2001)

    Google Scholar 

  37. Nasrollahi, K., Moeslund, T.B.: Super-resolution: a comprehensive survey. Mach. Vis. Appl. 25(6), 1423–1468 (2014)

    Article  Google Scholar 

  38. Odena, A., Dumoulin, V., Olah, C.: Deconvolution and checkerboard artifacts. Distill (2016). http://distill.pub/2016/deconv-checkerboard

  39. Pandey, R.K., Saha, N., Karmakar, S., Ramakrishnan, A.G.: MSCE: an edge preserving robust loss function for improving super-resolution algorithms. arXiv preprint arXiv:1809.00961 (2018)

  40. Ren, H., Kheradmand, A., El-Khamy, M., Wang, S., Bai, D., Lee, J.: Real-world super-resolution using generative adversarial networks. In: CVPR Workshops, pp. 1–8 (2020)

    Google Scholar 

  41. Sabour, S., Frosst, N., Hinton, G.E.: Dynamic routing between capsules. In: NeurIPS, pp. 3856–3866 (2017)

    Google Scholar 

  42. Sabour, S., Tagliasacchi, A., Yazdani, S., Hinton, G.E., Fleet, D.J.: Unsupervised part representation by flow capsules. In: Meila, M., Zhang, T. (eds.) ICML, vol. 139, pp. 9213–9223 (2021)

    Google Scholar 

  43. Salimans, T., Kingma, D.P.: Weight normalization: a simple reparameterization to accelerate training of deep neural networks. In: NeurIPS, pp. 901–909. Curran Associates, Inc. (2016)

    Google Scholar 

  44. Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: CVPR, pp. 1–8 (2016)

    Google Scholar 

  45. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2015)

    Google Scholar 

  46. Singh, M., Nagpal, S., Singh, R., Vatsa, M.: Dual directed capsule network for very low resolution image recognition. In: ICCV, pp. 1–8 (2019)

    Google Scholar 

  47. Sobel, I., Feldman, G.: A \(3\times 3\) Isotropic Gradient Operator for Image Processing (1968). Talk at the Stanford Artificial Project

    Google Scholar 

  48. Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.A.: Inception-v4, inception-resnet and the impact of residual connections on learning. In: AAAI, pp. 4278–4284 (2017)

    Google Scholar 

  49. Szegedy, C., et al.: Going deeper with convolutions. In: CVPR, pp. 1–8 (2015)

    Google Scholar 

  50. Timofte, R., et al.: NTIRE 2017 challenge on single image super-resolution: methods and results. In: CVPR Workshops, pp. 1110–1121 (2017)

    Google Scholar 

  51. Timofte, R., Gu, S., Wu, J., Van Gool, L.: NTIRE 2018 challenge on single image super-resolution: methods and results. In: CVPR Workshops, pp. 1–17 (2018)

    Google Scholar 

  52. Veit, A., Wilber, M.J., Belongie, S.: Residual networks behave like ensembles of relatively shallow networks. In: NeurIPS, pp. 550–558 (2016)

    Google Scholar 

  53. Wang, X., et al.: ESRGAN: enhanced super-resolution generative adversarial networks. In: ECCV, pp. 63–79 (2019)

    Google Scholar 

  54. Wang, Z., Liu, D., Yang, J., Han, W., Huang, T.: Deep networks for image super-resolution with sparse prior. In: ICCV, pp. 370–378 (2015)

    Google Scholar 

  55. Wang, Z., Simoncelli, E.P., Bovik, A.C.: Multiscale structural similarity for image quality assessment. In: The Thirty-Seventh Asilomar Conference on Signals, Systems Computers, vol. 2, pp. 1398–1402 (2003)

    Google Scholar 

  56. Wang, Z., Chen, J., Hoi, S.C.H.: Deep learning for image super-resolution: a survey. Trans. Pattern Anal. Mach. Intell. 43(10), 3365–3387 (2021)

    Article  Google Scholar 

  57. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  58. Xu, J., Zhao, Y., Dong, Y., Bai, H.: Fast and accurate image super-resolution using a combined loss. In: CVPR Workshops, pp. 1093–1099 (2017)

    Google Scholar 

  59. Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. Trans. Image Process. 19(11), 2861–2873 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  60. Yu, C., Zhu, X., Zhang, X., Wang, Z., Zhang, Z., Lei, Z.: HP-capsule: unsupervised face part discovery by hierarchical parsing capsule network. In: CVPR, pp. 4022–4031 (2022)

    Google Scholar 

  61. Yu, J., Fan, Y., Yang, J., Xu, N., Wang, X., Huang, T.S.: Wide Activation for Efficient and Accurate Image Super-Resolution. arXiv preprint arXiv:1808.08718 (2018)

  62. Zhang, K., Gu, S., Timofte, R.: NTIRE 2020 challenge on perceptual extreme super-resolution: methods and results. In: CVPR Workshops, pp. 1–10 (2020)

    Google Scholar 

  63. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: ECCV, pp. 1–8 (2018)

    Google Scholar 

  64. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: CVPR, pp. 2472–2481 (2018)

    Google Scholar 

  65. Zhao, H., Gallo, O., Frosio, I., Kautz, J.: Loss functions for image restoration with neural networks. IEEE Trans. Comput. Imaging 3(1), 47–57 (2017)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Computing, University of Campinas, Campinas, SP, 13083-852, Brazil

    George Corrêa de Araújo & Helio Pedrini

  2. Polytechnic School of the University of São Paulo, University of São Paulo, São Paulo, SP, 05508-010, Brazil

    Artur Jordão

Authors
  1. George Corrêa de Araújo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Artur Jordão
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Helio Pedrini
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Artur Jordão .

Editor information

Editors and Affiliations

  1. Federal University of São Carlos, São Carlos, Brazil

    Murilo C. Naldi

  2. Centro Universitario da FEI, São Bernardo do Campo, Brazil

    Reinaldo A. C. Bianchi

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

de Araújo, G.C., Jordão, A., Pedrini, H. (2023). Single Image Super-Resolution Based on Capsule Neural Networks. In: Naldi, M.C., Bianchi, R.A.C. (eds) Intelligent Systems. BRACIS 2023. Lecture Notes in Computer Science(), vol 14197. Springer, Cham. https://doi.org/10.1007/978-3-031-45392-2_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-45392-2_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-45391-5

  • Online ISBN: 978-3-031-45392-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation