Robust Local Light Field Synthesis via Occlusion-aware Sampling and Deep Visual Feature Fusion

Wenpeng Xing, Jie Chen*, Yi-Ke Guo

*Corresponding author for this work

Research output: Contribution to journalJournal articlepeer-review

3 Citations (Scopus)

Abstract

Novel view synthesis has attracted tremendous research attention recently for its applications in virtual reality and immersive telepresence. Rendering a locally immersive light field (LF) based on arbitrary large baseline RGB references is a challenging problem that lacks efficient solutions with existing novel view synthesis techniques. In this work, we aim at truthfully rendering local immersive novel views/LF images based on large baseline LF captures and a single RGB image in the target view. To fully explore the precious information from source LF captures, we propose a novel occlusion-aware source sampler (OSS) module which efficiently transfers the pixels of source views to the target view’s frustum in an occlusion-aware manner. An attention-based deep visual fusion module is proposed to fuse the revealed occluded background content with a preliminary LF into a final refined LF. The proposed source sampling and fusion mechanism not only helps to provide information for occluded regions from varying observation angles, but also proves to be able to effectively enhance the visual rendering quality. Experimental results show that our proposed method is able to render high-quality LF images/novel views with sparse RGB references and outperforms state-of-the-art LF rendering and novel view synthesis methods.
Original languageEnglish
Pages (from-to)408–420
Number of pages13
JournalMachine Intelligence Research
Volume20
Issue number3
Early online date17 Mar 2023
DOIs
Publication statusPublished - Jun 2023

Scopus Subject Areas

  • Artificial Intelligence
  • Signal Processing
  • Applied Mathematics
  • Control and Systems Engineering
  • Computer Vision and Pattern Recognition
  • Computer Networks and Communications
  • Computer Science Applications
  • Modelling and Simulation

User-Defined Keywords

  • Novel view synthesis
  • deep visual feature (DVF) fusion
  • light field (LF) imaging
  • multi-view stereo
  • occlusion sampling

Fingerprint

Dive into the research topics of 'Robust Local Light Field Synthesis via Occlusion-aware Sampling and Deep Visual Feature Fusion'. Together they form a unique fingerprint.

Cite this