Light Field Super-resolution via Attention-Guided Fusion of Hybrid Lenses

Jing Jin, Junhui Hou*, Jie Chen, Sam Kwong, Jingyi Yu

*Corresponding author for this work

Research output: Chapter in book/report/conference proceedingConference contributionpeer-review

4 Citations (Scopus)

Abstract

This paper explores the problem of reconstructing high-resolution light field (LF) images from hybrid lenses, including a high-resolution camera surrounded by multiple low-resolution cameras. To tackle this challenge, we propose a novel end-to-end learning-based approach, which can comprehensively utilize the specific characteristics of the input from two complementary and parallel perspectives. Specifically, one module regresses a spatially consistent intermediate estimation by learning a deep multidimensional and cross-domain feature representation; the other one constructs another intermediate estimation, which maintains the high-frequency textures, by propagating the information of the high-resolution view. We finally leverage the advantages of the two intermediate estimations via the learned attention maps, leading to the final high-resolution LF image. Extensive experiments demonstrate the significant superiority of our approach over state-of-the-art ones. That is, our method not only improves the PSNR by more than 2 dB, but also preserves the LF structure much better. To the best of our knowledge, this is the first end-to-end deep learning method for reconstructing a high-resolution LF image with a hybrid input. We believe our framework could potentially decrease the cost of high-resolution LF data acquisition and also be beneficial to LF data storage and transmission. The code is available at https://github.com/jingjin25/LFhybridSR-Fusion.

Original languageEnglish
Title of host publicationMM '20: Proceedings of the 28th ACM International Conference on Multimedia
PublisherAssociation for Computing Machinery (ACM)
Pages193-201
Number of pages9
ISBN (Electronic)9781450379885
DOIs
Publication statusPublished - 12 Oct 2020
Event28th ACM International Conference on Multimedia, MM 2020 - Virtual, Online, United States
Duration: 12 Oct 202016 Oct 2020

Publication series

NameProceedings of ACM International Conference on Multimedia

Conference

Conference28th ACM International Conference on Multimedia, MM 2020
Country/TerritoryUnited States
CityVirtual, Online
Period12/10/2016/10/20

Scopus Subject Areas

  • Software
  • Computer Graphics and Computer-Aided Design
  • Human-Computer Interaction

User-Defined Keywords

  • attention
  • deep learning
  • hybrid imaging system
  • light field
  • super-resolution

Fingerprint

Dive into the research topics of 'Light Field Super-resolution via Attention-Guided Fusion of Hybrid Lenses'. Together they form a unique fingerprint.

Cite this