Real-Time Scene Analysis for 3D Interaction

PhD Thesis. Work done at Telecom Paris in collaboration with Ayotle SAS.
Collaborators: Prof. Tamy Boubekeur, Dr. Jose Alonso Ybanez Zepeda

Keywords: Visual scene analysis, Commodity depth sensors, Geometric analysis, High level abstraction

This PhD thesis focuses on the problem of visual scene analysis captured by commodity depth sensors to convert their data into high level understanding of the scene. It explores the use of 3D geometry analysis tools on visual depth data in terms of enhancement, registration and consolidation. In particular, we aim to show how shape abstraction can generate lightweight representations of the data for fast analysis with low hardware requirements. This last property is important as one of our goals is to design algorithms suitable for live embedded operation in e.g., wearable devices, smartphones or mobile robots. The context of this thesis is the live operation of 3D interaction on a mobile device, which raises numerous issues including placing 3D interaction zones with relation to real surrounding objects, tracking the interaction zones in space when the sensor moves and providing a meaningful and understandable experience to non-expert users. Towards solving these problems, we make contributions where scene abstraction leads to fast and robust sensor localization as well as efficient frame data representation, enhancement and consolidation. While simple geometric surface shapes are not as faithful as heavy point sets or volumes to represent observed scenes, we show that they are an acceptable approximation and their light weight makes them well balanced between accuracy and performance.

Plane Pair Matching for Efficient 3D View Registration

Work done during my PhD at Telecom Paris in collaboration with Ayotle SAS.
Collaborators: Prof. Tamy Boubekeur, Dr. Jose Alonso Ybanez Zepeda

Keywords: RGB-D images, Registration, Localization, Plane pairs, Scene structure

We present a novel method to estimate the motion matrix between overlapping pairs of 3D views in the context of indoor scenes. We use the Manhattan world assumption to introduce lightweight geometric constraints under the form of planes into the problem, which reduces complexity by taking into account the structure of the scene. In particular, we define a stochastic framework to categorize planes as vertical or horizontal and parallel or non-parallel. We leverage this classification to match pairs of planes in overlapping views with point-of-view agnostic structural metrics. We propose to split the motion computation using the classification and estimate separately the rotation and translation of the sensor, using a quadric minimizer. We validate our approach on a toy example and present quantitative experiments on a public RGB-D dataset, comparing against recent state-of-the-art methods. Our evaluation shows that planar constraints only add low computational overhead while improving results in precision when applied after a prior coarse estimate. We conclude by giving hints towards extensions and improvements of current results.

  A. Kaiser, J.A. Ybanez Zepeda and T. Boubekeur:
Plane Pair Matching for Efficient 3D View Registration
ArXiv 2020     

  A. Kaiser, J.A. Ybanez Zepeda, T. Boubekeur and A. Courteville:
Method for Registering Depth Images
WO 2020/169381 A1 (priority date 2019-02-22)   

Proxy Clouds for Live RGB-D Stream Processing and Consolidation

Work done during my PhD at Telecom Paris in collaboration with Ayotle SAS.
Collaborators: Prof. Tamy Boubekeur, Dr. Jose Alonso Ybanez Zepeda

Keywords: RGB-D images, Scene analysis, Geometric analysis, 3D point cloud, 3D geometric primitives, planes, Data accumulation, High level superstructure, Live processing, Depth improvement, Data reinforcement, Scene reconstruction, 3D data compression

We propose a new multiplanar superstructure for unified real-time processing of RGB-D data. Modern RGB-D sensors are widely used for indoor 3D capture, with applications ranging from modeling to robotics, through augmented reality. Nevertheless, their use is limited by their low resolution, with frames often corrupted with noise, missing data and temporal inconsistencies. Our approach, named Proxy Clouds, consists in generating and updating through time a single set of compact local statistics parameterized over detected planar proxies, which are fed from raw RGB-D data. Proxy Clouds provide several processing primitives, which improve the quality of the RGB-D stream on-the-fly or lighten further operations. Experimental results confirm that our light weight analysis framework copes well with embedded execution as well as moderate memory and computational capabilities compared to state-of-the-art methods. Processing of RGB-D data with Proxy Clouds includes noise and temporal flickering removal, hole filling and resampling. As a substitute of the observed scene, our proxy cloud can additionally be applied to compression and scene reconstruction. We present experiments performed with our framework in indoor scenes of different natures within a recent open RGB-D dataset.

  A. Kaiser, J.A. Ybanez Zepeda and T. Boubekeur:
Proxy Clouds for RGB-D Stream Processing: A Preview
EUROGRAPHICS 2017 - Posters Program     

  EUROGRAPHICS 2017, April 24-28, Lyon, France  

  A. Kaiser, J.A. Ybanez Zepeda and T. Boubekeur:
Proxy Clouds for RGB-D Stream Processing: An Insight
ACM SIGGRAPH 2017 - Talks Program       

  ACM SIGGRAPH 2017, July 30 - August 3, Los Angeles, CA, USA  

  A. Kaiser, J.A. Ybanez Zepeda and T. Boubekeur:
Proxy Clouds for Live RGB-D Stream Processing and Consolidation
European Conference on Computer Vision (ECCV) 2018       
(h5-index = 137 at time of publication, 31.8% acceptance)

  ECCV 2018, September 8 - 14, Munich, Germany  

  A. Kaiser, J.A. Ybanez Zepeda and T. Boubekeur:
Geometric Proxies for Live RGB-D Stream Enhancement and Consolidation
ArXiv 2020 (extension paper)     

A Survey of Simple Geometric Primitives Detection Methods for Captured 3D Data

Work done during my PhD at Telecom Paris in collaboration with Ayotle SAS.
Collaborators: Prof. Tamy Boubekeur, Dr. Jose Alonso Ybanez Zepeda

Keywords: 3D data, 3D geometric primitives, Shape analysis, Shape abstraction, Geometric modeling, Data fitting

The amount of captured 3D data is continuously increasing, with the democratization of consumer depth cameras, the development of modern multi-view stereo capture setups and the rise of single-view 3D capture based on machine learning. The analysis and representation of this ever growing volume of 3D data, often corrupted with acquisition noise and reconstruction artifacts, is a serious challenge at the frontier between computer graphics and computer vision. To that end, segmentation and optimization are crucial analysis components of the shape abstraction process, which can themselves be greatly simplified when performed on lightened geometric formats. In this survey, we review the algorithms which extract simple geometric primitives from raw dense 3D data. After giving an introduction to these techniques, from the acquisition modality to the underlying theoretical concepts, we propose an application-oriented characterization, designed to help select an appropriate method based on one's application needs, and compare recent approaches. We conclude by giving hints for how to evaluate these methods and a set of research challenges to be explored.

  A. Kaiser, J.A. Ybanez Zepeda and T. Boubekeur:
A Survey of Simple Geometric Primitives Detection Methods for Captured 3D Data
Computer Graphics Forum, 38(1):167-196, February 2019 (first published July 2018).     
(h5-index = 49 at time of publication)

  EUROGRAPHICS 2019, May 6-10, Genova, Italy  

Robust 3D Object Detection and Tracking on Mobile Devices

Master's Thesis. Work done while at Wikitude GmbH in collaboration with TU Wien.
Collaborators: Nicolas Goell, Markus Eder

Keywords: Augmented reality, Mobile devices, Scene tracking, Simultaneous Localization And Mapping (SLAM), RGB images, Point features, Edge features, Keyframes, SIMD on embedded CPU (ARM NEON).

Creation of a visual detection and tracking system based on Simultaneous Localization and Mapping (SLAM) for three dimensional objects on mobile devices, for augmented reality applications. Study of the state of the art and implementation of the detector in C++, inspired by existing methods.

DTIAtlasBuilder

Work done as an intern at the NIRAL within the University of North Carolina at Chapel Hill.
Collaborators: Dr. Martin Styner, Francois Budin

Keywords: Brain imaging, Diffusion Tensor Images (DTI), Magnetic Resonance Imaging (MRI), Tractography, 3D voxel registration, Qt, ITK, Graphical user interface, Cross-platform, 3DSlicer.

Development of a user interface for the creation of an atlas image of Diffusion Tensor Images. Cross-platform implementation of the interface in C++ and Python with Qt and ITK to execute the image processing pipeline. Integration to 3DSlicer as an extension.
Several contributions to open-source projects.

  A. R. Verde, F. Budin, J.B. Berger, A. Gupta, M. Farzinfar, A. Kaiser, M. Ahn, H. Johnson, J. Matsui, H. C. Hazlett, A. Sharma, C. Goodlett, Y. Shi, S. Gouttard, C. Vachet, J. Piven, H. Zhu, G. Gerig and M. Styner: UNC-Utah NA-MIC framework for DTI fiber tract analysis
Frontiers in Neuroinformatics, January 2014  
(h5-index = 26 at time of publication)

  2013 NA-MIC All Hands Meeting, Jan 7-11, Salt Lake City, UT, USA