Beyond the Lens: Leveraging Neural Radiance Fields and Generative AI for Crossmedia 3D Design

Recent AI-based advancements have significantly impacted the domain of 3D digital imagery. The intersection of photorealistic spatial reconstruction and synthetic image generation has blurred the boundaries between indexical lens-based digital images and synthetically created digital photorealistic 3D content. Recent advancements in neural radiance fields, structure from motion, and simultaneous localization and mapping (SLAM) techniques have enabled the reconstruction of real- life objects and locations with unprecedented accuracy and visual fidelity. Concurrently, diffusion-based Generative AI models have exponentially increased the potential for generating still images, videos, and simple 3D assets. This convergence has given rise to a rhizomatic entanglement of styles, behaviors, and capture settings originating from lens-based cameras and their virtual counterparts. The resulting porosity of styles and aesthetics has created opportunities for innovative crossmedia image sensibilities, bridging the conceptual space between visual/digital twins and synthetic generations of real-life, fictional, and hybrid styles, characters, objects, and locations. This presentation showcases a series of work-in-progress projects that aim to investigate the potential of novel methods for capturing, processing, and displaying 3D reconstructions/generations of real-life locations and objects. The focus will be on leveraging 3D Gaussian Splatting, SLAM-enhanced digital twins, and large LED display volumes to preserve cultural heritage sites in Hong Kong and Poland. Additionally, another project will explore the creation of immersive experiences based on spatial scans of complex natural environments as well as photorealistic synthetic 3D assets combined in a unified game engine-based real-time environment. This contribution serves as both a technical ideation and a cultural intervention in emerging technologies. It reflects on the conceptual and technical challenges faced when working with rapidly evolving spatial 3D media. Based on the presented works and numerous educational activities undertaken in the discussed domain, this contribution investigates how the space between visual/digital twins and synthetic generations can be leveraged to create new crossmedia image sensibilities, thereby contributing to the broader discussion of synthography, and the role of camera and cinematography in today's media landscape. Additionally, by discussing these technological advancements through an artistic and educational lens, this contribution explores the potential of collaboration between human artists and AI tools, while also addressing ethical considerations in ownership and accessibility.