How to reduce TOF errors in AR glasses

In this blog, I will describe how we reduced the noise of the Time-Of-Flight sensor in our AR glasses prototype.

Types of noise
- systematic noise
   note: caused by imperfect sinusoidal modulation
- random noise
   note: by shot noise. use bilateral filtering

Motion artifacts reduction
note: when motion is observed on a target object, we have motion artifacts observed in the tof sensor.  This happens when TOF measurement is recorded sequentially.  And, this causes doppler effects.

fix:
- use Plus and Minus rules
   -- reference:
       1) "Time of flight motion compensation revisited"  (2014)
       2) "Time of flight cameras: Principles, Methods and Applications" (2012)


Physics-based MPI reduction

fix:
- use 2K+1 frequency measurements for K inferencing paths in absence of noise.


Per-pixel temporal processing of raw ToF measurements

fix:
- matrix pencil method
- Prong's method
- onthogonal matching method
- EPIRIT / MUSIC
- atomic norm regularization
- light transport model with sparse & low rank components
- phaser imaging

reference:
- "Signal processing for time-of-flight imaging sensors: An introduction to inverse problems in computational 3-d imaging" (2016)
- "Resolving multipath interference in kinetc: An inverse problem approach." (2016)
- "Recent advances in transient imaging: A computer graphics and vision perspective" (2017)
- "SRA: fast removal of general multipath for tof sensors." (2014)
- "Phasor imaging: A generalization of correlation-based time-of-flight imaging"  (2015)

Learning-based MPI reduction

fix:
- use an encoder to learn a mapping from captured ToF measurements to a feature representation of MPI corrupted path.
- combine it with a simulated, directed ToF measurements to train a decoder, so it can produce MPI corrected depth maps.

- use a KAKU robot and structured light to capture ToF measurements with registered GT depth.
- then, train two neural networks to correct depth and refine edges using geodesic filtering.

- use transient rendering to synthesize a training dataset with realistic shot noise.
- then, generate measurements from ToF sensors with random modulation path.

reference:
- ""DeepToF: Off the self real-time correction of multipath interference in time-of-flight imaging" (2017)
- "Automatic learning to remove multipath distortions in time-of-flight range images for a robotic arm setup." (2016)
- "Recent advances in transient imaging: A computer graphics and vision perspective" (2017)
- "A framework for transient rendering." (2014)


Additional notes will be added in the near future, as we make more progress.