Tensor Holography V2 improves over V1 by synthesizing 3D phase-only holograms end-to-end with deep double phase encoding, a learning-based phase-only encoding that avoids hand-tuning filters. The V2 pipeline leverages the layered depth images as 3D inputs to achieve more realistic occlusion reproduction and overcome the data challenges in using sihoulet-mask layered-based method [Zhang et al. 2017] with dense depth partition. It is robust to artifacts and misalignments in the captured depth map for real-world data (i.e. content reconstructed by NeRF as shown above [Mildenhall et al. 2020]).

Computer-generated holography (CGH) provides volumetric control of coherent wavefront and is fundamental to applications such as volumetric 3D displays, lithography, neural photostimulation, and optical/acoustic trapping. Recently, deep learning-based methods emerged as promising computational paradigms for CGH synthesis that overcome the quality-runtime tradeoff in conventional simulation/optimization-based methods. Yet, the quality of the predicted hologram is intrinsically bounded by the dataset's quality. Here we introduce a new hologram dataset, MIT-CGH-4K-V2, that uses a layered depth image as a data-efficient volumetric 3D input and a two-stage supervised+unsupervised training protocol for direct synthesis of high-quality 3D phase-only holograms. The proposed system also corrects vision aberration, allowing customization for end-users. We experimentally show photorealistic 3D holographic projections and discuss relevant spatial light modulator calibration procedures. Our method runs in real-time on a consumer GPU and 5 FPS on an iPhone 13 Pro, promising drastically enhanced performance for the applications above.