Tensor Holography V2:
End-to-end Learning of 3D Phase-only Holograms for
Holographic Display

Light: Science & Applications 2022

Liang Shi1,2,✉     Beichen Li1,2     Wojciech Matusik1,2,✉
1MIT CSAIL      2MIT EECS      Corresponding Author
Tensor Holography V2 improves over V1 by synthesizing 3D phase-only holograms end-to-end with deep double phase encoding, a learning-based phase-only encoding that avoids hand-tuning filters. The V2 pipeline leverages the layered depth images as 3D inputs to achieve more realistic occlusion reproduction and overcome the data challenges in using sihoulet-mask layered-based method [Zhang et al. 2017] with dense depth partition. It is robust to artifacts and misalignments in the captured depth map for real-world data (i.e. content reconstructed by NeRF as shown above [Mildenhall et al. 2020]).


Computer-generated holography (CGH) provides volumetric control of coherent wavefront and is fundamental to applications such as volumetric 3D displays, lithography, neural photostimulation, and optical/acoustic trapping. Recently, deep learning-based methods emerged as promising computational paradigms for CGH synthesis that overcome the quality-runtime tradeoff in conventional simulation/optimization-based methods. Yet, the quality of the predicted hologram is intrinsically bounded by the dataset's quality. Here we introduce a new hologram dataset, MIT-CGH-4K-V2, that uses a layered depth image as a data-efficient volumetric 3D input and a two-stage supervised+unsupervised training protocol for direct synthesis of high-quality 3D phase-only holograms. The proposed system also corrects vision aberration, allowing customization for end-users. We experimentally show photorealistic 3D holographic projections and discuss relevant spatial light modulator calibration procedures. Our method runs in real-time on a consumer GPU and 5 FPS on an iPhone 13 Pro, promising drastically enhanced performance for the applications above.


End-to-end Learning of 3D Phase-only Holograms for Holographic Display
Liang Shi, Beichen Li, Wojciech Matusik
Light: Science & Applications 2022
[Paper]  [Code]  [Dataset]  [BibTeX]

Towards Real-time Photorealistic 3D Holography with Deep Neural Networks
Liang Shi, Beichen Li, Changil Kim, Petr Kellnhofer, Wojciech Matusik
Nature 2021
[Paper]  [Code]  [Dataset]  [BibTeX]

Video (Coming soon)