Microlens Amplitude Masks for Flying Pixel Removal in Time-of-Flight Imaging
Princeton Docket # 21-3789
Flying Pixels (FPs) are common artifacts in time-of-flight (ToF) imaging that arise at object boundaries, where light from both foreground and background is mixed, leading to inaccurate depth measurements and negatively impacting 3D vision tasks. The Mask-ToF approach addresses this issue by learning a microlens-level occlusion mask that customizes the sub-aperture for each sensor pixel. This mask modulates the integration of foreground and background light on a per-pixel basis, effectively encoding scene geometry directly into the ToF measurements. By using a differentiable ToF simulator, Mask-ToF trains a refinement network that decodes this information, producing high-fidelity depth reconstructions with significantly reduced FP counts.
The technology has been validated through both simulated datasets and an experimental prototype, demonstrating that the optimized mask pattern can halve FP counts without requiring retraining, thereby generalizing well to real-world data. This innovation can be easily integrated into the manufacturing process of ToF devices, as the optimized mask can be lithographically fabricated and directly adhered to the sensor surface, with minimal impact on form factor and cost. Additionally, the computational decoding of the masked depth signal can be incorporated into on-board software for real-time depth reconstruction.
Applications
Advantages
Stage of development
A prototype was lithographically manufactured and integrated into an optical relay system for a Helios Flex time-of-flight camera. Depth measurements taken with this optimized mask demonstrated a significant reduction in flying pixel count while maintaining a high signal-to-noise ratio, outperforming hand-crafted naïve masks and the no-mask condition.
Publication https://openaccess.thecvf.com/content/CVPR2021/papers/Chugunov_Mask-ToF_Learning_Microlens_Masks_for_Flying_Pixel_Correction_in_Time-of-Flight_CVPR_2021_paper.pdf
Inventors
Felix Heide Ph.D. is a Professor of Computer Science at Princeton University, where he leads the Princeton Computational Imaging Lab and serves as Head of AI at Torc Robotics, specializing in full autonomy for self-driving trucks. His research focuses on advanced imaging and computer vision techniques that enhance the capabilities of various sensors, addressing challenges in extreme conditions and integrating insights from optics, machine learning, and optimization.
Seung-Hwan Baek Ph.D. completed his postdoctoral fellowship at the Princeton Computational Imaging Lab and is now a faculty member at POSTECH.
Qiang Fu Ph.D. is a research scientist at King Abdullah University of Science and Technology. His research focuses on computational imaging, optical system design and nano-fabrication.
Wolfgang Heidrich Ph.D. is a research scientist at King Abdullah University of Science and Technology. His core research interests are in computational imaging and display.
Ilya Chungunov is a graduate student at Princeton University conducting research on computational imaging in the Princeton Computational Imaging Lab.
Intellectual Property & Development status
Patent protection is pending. Princeton is currently seeking commercial partners for the further development and commercialization of this opportunity.
Contact
Prabhpreet Gill
Princeton University Office of Technology Licensing • (609)258-3653 • psgill@princeton.edu