Abstract
We present MVSPlenOctree, a novel approach that can efficiently reconstruct radiance fields for view synthesis. Unlike previous scene-specific radiance fields reconstruction methods, we present a generic pipeline that can efficiently reconstruct 360-degree-renderable radiance fields via multi-view stereo (MVS) inference from tens of sparse-spread out images. Our approach leverages variance-based statistic features for MVS inference, and combines this with image based rendering and volume rendering for radiance field reconstruction. We first train a MVS Machine for reasoning scene's density and appearance. Then, based on the spatial hierarchy of the PlenOctree and coarse-to-fine dense sampling mechanism, we design a robust and efficient sampling strategy for PlenOctree reconstruction, which handles occlusion robustly. A 360-degree-renderable radiance fields can be reconstructed in PlenOctree from MVS Machine in an efficient single forward pass. We trained our method on real-world DTU, LLFF datasets, and synthetic datasets. We validate its generalizability by evaluating on the test set of DTU dataset which are unseen in training. In summary, our radiance field reconstruction method is both efficient and generic, a coarse 360-degree-renderable radiance field can be reconstructed in seconds and a dense one within minutes. Please visit the project page for more details: https://derry-xing.github.io/projects/MVSPlenOctree.
Original language | English |
---|---|
Title of host publication | MM '22: Proceedings of the 30th ACM International Conference on Multimedia |
Publisher | Association for Computing Machinery (ACM) |
Pages | 5114–5122 |
Number of pages | 9 |
ISBN (Print) | 9781450392037 |
DOIs | |
Publication status | Published - 10 Oct 2022 |
Event | 30th ACM International Conference on Multimedia, MM 2022 - Lisboa, Portugal Duration: 10 Oct 2022 → 14 Oct 2022 https://dl.acm.org/doi/proceedings/10.1145/3503161 |
Conference
Conference | 30th ACM International Conference on Multimedia, MM 2022 |
---|---|
Country/Territory | Portugal |
City | Lisboa |
Period | 10/10/22 → 14/10/22 |
Internet address |
User-Defined Keywords
- neural radiance fields
- novel view synthesis
- multi-view stereo