TY - JOUR
T1 - Deep Spatial-angular Regularization for Light Field Imaging, Denoising, and Super-resolution
AU - Guo, Mantang
AU - Hou, Junhui
AU - Jin, Jing
AU - Chen, Jie
AU - Chau, Lap-Pui
N1 - Funding Information:
This work was supported in part by the Hong Kong Research Grants Coun-cil under grants 9048123 (CityU 21211518) and 9042820 (CityU 11219019), and in part by the Basic Research General Program of Shenzhen Municipalityunder grants JCYJ20190808183003968.
Publisher copyright:
© 2021 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
PY - 2022/10/1
Y1 - 2022/10/1
N2 - Coded aperture is a promising approach for capturing the 4-D light field (LF), in which the 4-D data are compressively modulated into 2-D coded measurements that are further decoded by reconstruction algorithms. The bottleneck lies in the reconstruction algorithms, resulting in rather limited reconstruction quality. To tackle this challenge, we propose a novel learning-based framework for the reconstruction of high-quality LFs from acquisitions via learned coded apertures. The proposed method incorporates the measurement observation into the deep learning framework elegantly to avoid relying entirely on data-driven priors for LF reconstruction. Specifically, we first formulate the compressive LF reconstruction as an inverse problem with an implicit regularization term. Then, we construct the regularization term with a deep efficient spatial-angular separable convolutional sub-network in the form of local and global residual learning to comprehensively explore the signal distribution free from the limited representation ability and inefficiency of deterministic mathematical modeling. Furthermore, we extend this pipeline to LF denoising and spatial super-resolution, which could be considered as variants of coded aperture imaging equipped with different degradation matrices. Extensive experimental results demonstrate that the proposed methods outperform state-of-the-art approaches to a significant extent both quantitatively and qualitatively, i.e., the reconstructed LFs not only achieve much higher PSNR/SSIM but also preserve the LF parallax structure better on both real and synthetic LF benchmarks. The code will be publicly available at https://github.com/MantangGuo/DRLF.
AB - Coded aperture is a promising approach for capturing the 4-D light field (LF), in which the 4-D data are compressively modulated into 2-D coded measurements that are further decoded by reconstruction algorithms. The bottleneck lies in the reconstruction algorithms, resulting in rather limited reconstruction quality. To tackle this challenge, we propose a novel learning-based framework for the reconstruction of high-quality LFs from acquisitions via learned coded apertures. The proposed method incorporates the measurement observation into the deep learning framework elegantly to avoid relying entirely on data-driven priors for LF reconstruction. Specifically, we first formulate the compressive LF reconstruction as an inverse problem with an implicit regularization term. Then, we construct the regularization term with a deep efficient spatial-angular separable convolutional sub-network in the form of local and global residual learning to comprehensively explore the signal distribution free from the limited representation ability and inefficiency of deterministic mathematical modeling. Furthermore, we extend this pipeline to LF denoising and spatial super-resolution, which could be considered as variants of coded aperture imaging equipped with different degradation matrices. Extensive experimental results demonstrate that the proposed methods outperform state-of-the-art approaches to a significant extent both quantitatively and qualitatively, i.e., the reconstructed LFs not only achieve much higher PSNR/SSIM but also preserve the LF parallax structure better on both real and synthetic LF benchmarks. The code will be publicly available at https://github.com/MantangGuo/DRLF.
KW - Light field
KW - coded aperture
KW - deep learning
KW - optimization
KW - observation model
KW - denoising
KW - spatial super-resolution
KW - depth
UR - http://www.scopus.com/inward/record.url?scp=85111008796&partnerID=8YFLogxK
U2 - 10.1109/TPAMI.2021.3087485
DO - 10.1109/TPAMI.2021.3087485
M3 - Journal article
AN - SCOPUS:85111008796
SN - 0162-8828
VL - 44
SP - 6094
EP - 6110
JO - IEEE Transactions on Pattern Analysis and Machine Intelligence
JF - IEEE Transactions on Pattern Analysis and Machine Intelligence
IS - 10
ER -