RRM: Relightable assets using Radiance guided Material extraction

We propose RRM, a method that can extract the materials, geometry, and environment lighting of a scene even in the presence of highly reflective objects. Novel view synthesis has seen tremendous improvements over the last few years thanks to the rise of neural representations of light fields, shifting the paradigm for the acquisition of 3D data from photographs. However, these radiance-based approaches typically lack the ability to synthesize views under new lighting conditions. Recent efforts tackle the problem via the extraction of physically-based parameters that can then be rendered under arbitrary lighting, but they are limited in the range of scenes they can handle, usually mishandling glossy scenes. We design a physically-aware radiance fields representation to supervise the diffuse and view-dependent components of a physically-based module that uses Multiple Importance Sampling to grasp the complex behavior of glossy indirect lighting as well as feed our expressive environment light structure based on Pyramids of Laplacians. We demonstrate that our contributions outperform the state-of-the-art on parameter retrieval tasks, leading to high-fidelity relighting and novel view synthesis on surfacic scenes. (more)

  D. Gomez, J. Philip, A. Kaiser, and É. Michel:
RRM: Relightable Assets using Radiance Guided Material Extraction
CGI 2024

  CGI 2024, July 1-5, Geneva, Switzerland  

ControlMat: Controlled Generative Approach to Material Capture

Material reconstruction from a photograph is a key component of 3D content creation democratization. We propose to formulate this ill-posed problem as a controlled synthesis one, leveraging the recent progress in generative deep networks. We present ControlMat, a method which, given a single photograph with uncontrolled illumination as input, conditions a diffusion model to generate plausible, tileable, high-resolution physically-based digital materials. We carefully analyze the behavior of diffusion models for multi-channel outputs, adapt the sampling process to fuse multi-scale information and introduce rolled diffusion to enable both tileability and patched diffusion for high-resolution outputs. Our generative approach further permits exploration of a variety of materials which could correspond to the input image, mitigating the unknown lighting conditions. We show that our approach outperforms recent inference and latent-space-optimization methods, and carefully validate our diffusion process design choices. (more)

  G. Vecchio, R. Martin, A. Roullier, A. Kaiser, R. Rouffet, V. Deschaintre, and T. Boubekeur:
ControlMat: Controlled Generative Approach to Material Capture
Arxiv 2023   
(accepted to ACM Transactions on Graphics, invited to SIGGRAPH Asia 2024)

Semantic Shape Editing with Parametric Implicit Templates

We propose a semantic shape editing method to edit 3D triangle meshes using parametric implicit surface templates, benefiting from the many advantages offered by analytical implicit representations , such as infinite resolution and boolean or blending operations. We propose first a template fitting method to optimize its parameters to best capture the input mesh. For subsequent template edits, our novel mesh deformation method allows tracking the template’s 0-set even when featuring anisotropic stretch and/or local volume change. We make few assumptions on the template implicit fields and only strictly require continuity. We demonstrate applications to interactive semantic shape editing and semantic mesh retargeting. (more)

  U. Kusupati, M. Gaillard, J-M. Thiery, A. Kaiser:
Semantic Shape Editing with Parametric Implicit Templates
SIGGRAPH 2024

  SIGGRAPH 2024, July 28 - August 1, Denver, Colorado USA  

  A. Kaiser, V. Krs, T. Groueix, T. Boubekeur, P. Gueth, M. Gaillard, and M. Gadelha:
Modeling shapes using differentiable signed distance functions
US 2023/0274040 A1 (priority date 2022-02-28)   

MaterIA: Single Image High-Resolution Material Capture in the Wild

We propose a hybrid method to reconstruct a physically-based spatially varying BRDF from a single high resolution picture of an outdoor surface captured under natural lighting conditions with any kind of camera device. Relying on both deep learning and explicit processing, our PBR material acquisition handles the removal of shades, projected shadows and specular highlights present when capturing a highly irregular surface and enables to properly retrieve the underlying geometry. To achieve this, we train two cascaded U-Nets on physically-based materials, rendered under various lighting conditions, to infer the spatially-varying albedo and normal maps. Our network processes relatively small image tiles (512x512 pixels) and we propose a solution to handle larger image resolutions by solving a Poisson system across these tiles. We complete this pipeline with analytical solutions to reconstruct height, roughness and ambient occlusion. (more)

  R. Martin, A. Roullier, R. Rouffet, A. Kaiser, and T. Boubekeur:
MaterIA: Single Image High-Resolution Material Capture in the Wild
EUROGRAPHICS 2022

  EUROGRAPHICS 2022, April 25-29, Reims, France  

  T. Boubekeur, A. Kaiser, R. Martin, R. Rouffet, and A. Roullier:
Learning material reconstruction from a single image
US 11,670,042 (priority date 2021-03-15)   

Real-Time Scene Analysis for 3D Interaction

PhD Thesis. Work done at Telecom Paris in collaboration with Ayotle SAS.
Collaborators: Prof. Tamy Boubekeur, Dr. Jose Alonso Ybanez Zepeda

Keywords: Visual scene analysis, Commodity depth sensors, Geometric analysis, High level abstraction

This PhD thesis focuses on the problem of visual scene analysis captured by commodity depth sensors to convert their data into high level understanding of the scene. It explores the use of 3D geometry analysis tools on visual depth data in terms of enhancement, registration and consolidation. In particular, we aim to show how shape abstraction can generate lightweight representations of the data for fast analysis with low hardware requirements. This last property is important as one of our goals is to design algorithms suitable for live embedded operation in e.g., wearable devices, smartphones or mobile robots. The context of this thesis is the live operation of 3D interaction on a mobile device, which raises numerous issues including placing 3D interaction zones with relation to real surrounding objects, tracking the interaction zones in space when the sensor moves and providing a meaningful and understandable experience to non-expert users. Towards solving these problems, we make contributions where scene abstraction leads to fast and robust sensor localization as well as efficient frame data representation, enhancement and consolidation. While simple geometric surface shapes are not as faithful as heavy point sets or volumes to represent observed scenes, we show that they are an acceptable approximation and their light weight makes them well balanced between accuracy and performance. (more)

Plane Pair Matching for Efficient 3D View Registration

Work done during my PhD at Telecom Paris in collaboration with Ayotle SAS.
Collaborators: Prof. Tamy Boubekeur, Dr. Jose Alonso Ybanez Zepeda

Keywords: RGB-D images, Registration, Localization, Plane pairs, Scene structure

We present a novel method to estimate the motion matrix between overlapping pairs of 3D views in the context of indoor scenes. We use the Manhattan world assumption to introduce lightweight geometric constraints under the form of planes into the problem, which reduces complexity by taking into account the structure of the scene. In particular, we define a stochastic framework to categorize planes as vertical or horizontal and parallel or non-parallel. We leverage this classification to match pairs of planes in overlapping views with point-of-view agnostic structural metrics. We propose to split the motion computation using the classification and estimate separately the rotation and translation of the sensor, using a quadric minimizer. We validate our approach on a toy example and present quantitative experiments on a public RGB-D dataset, comparing against recent state-of-the-art methods. Our evaluation shows that planar constraints only add low computational overhead while improving results in precision when applied after a prior coarse estimate. We conclude by giving hints towards extensions and improvements of current results. (more)

  A. Kaiser, J.A. Ybanez Zepeda and T. Boubekeur:
Plane Pair Matching for Efficient 3D View Registration
ArXiv 2020     

  A. Kaiser, J.A. Ybanez Zepeda, T. Boubekeur and A. Courteville:
Method for Registering Depth Images
WO 2020/169381 A1 (priority date 2019-02-22)   

Proxy Clouds for Live RGB-D Stream Processing and Consolidation

Work done during my PhD at Telecom Paris in collaboration with Ayotle SAS.
Collaborators: Prof. Tamy Boubekeur, Dr. Jose Alonso Ybanez Zepeda

Keywords: RGB-D images, Scene analysis, Geometric analysis, 3D point cloud, 3D geometric primitives, planes, Data accumulation, High level superstructure, Live processing, Depth improvement, Data reinforcement, Scene reconstruction, 3D data compression

We propose a new multiplanar superstructure for unified real-time processing of RGB-D data. Modern RGB-D sensors are widely used for indoor 3D capture, with applications ranging from modeling to robotics, through augmented reality. Nevertheless, their use is limited by their low resolution, with frames often corrupted with noise, missing data and temporal inconsistencies. Our approach, named Proxy Clouds, consists in generating and updating through time a single set of compact local statistics parameterized over detected planar proxies, which are fed from raw RGB-D data. Proxy Clouds provide several processing primitives, which improve the quality of the RGB-D stream on-the-fly or lighten further operations. Experimental results confirm that our light weight analysis framework copes well with embedded execution as well as moderate memory and computational capabilities compared to state-of-the-art methods. Processing of RGB-D data with Proxy Clouds includes noise and temporal flickering removal, hole filling and resampling. As a substitute of the observed scene, our proxy cloud can additionally be applied to compression and scene reconstruction. We present experiments performed with our framework in indoor scenes of different natures within a recent open RGB-D dataset. (more)

  A. Kaiser, J.A. Ybanez Zepeda and T. Boubekeur:
Proxy Clouds for RGB-D Stream Processing: A Preview
EUROGRAPHICS 2017 - Posters Program     

  EUROGRAPHICS 2017, April 24-28, Lyon, France  

  A. Kaiser, J.A. Ybanez Zepeda and T. Boubekeur:
Proxy Clouds for RGB-D Stream Processing: An Insight
ACM SIGGRAPH 2017 - Talks Program       

  ACM SIGGRAPH 2017, July 30 - August 3, Los Angeles, CA, USA  

  A. Kaiser, J.A. Ybanez Zepeda and T. Boubekeur:
Proxy Clouds for Live RGB-D Stream Processing and Consolidation
European Conference on Computer Vision (ECCV) 2018       
(h5-index = 137 at time of publication, 31.8% acceptance)

  ECCV 2018, September 8 - 14, Munich, Germany  

  A. Kaiser, J.A. Ybanez Zepeda and T. Boubekeur:
Geometric Proxies for Live RGB-D Stream Enhancement and Consolidation
ArXiv 2020 (extension paper)     

A Survey of Simple Geometric Primitives Detection Methods for Captured 3D Data

Work done during my PhD at Telecom Paris in collaboration with Ayotle SAS.
Collaborators: Prof. Tamy Boubekeur, Dr. Jose Alonso Ybanez Zepeda

Keywords: 3D data, 3D geometric primitives, Shape analysis, Shape abstraction, Geometric modeling, Data fitting

The amount of captured 3D data is continuously increasing, with the democratization of consumer depth cameras, the development of modern multi-view stereo capture setups and the rise of single-view 3D capture based on machine learning. The analysis and representation of this ever growing volume of 3D data, often corrupted with acquisition noise and reconstruction artifacts, is a serious challenge at the frontier between computer graphics and computer vision. To that end, segmentation and optimization are crucial analysis components of the shape abstraction process, which can themselves be greatly simplified when performed on lightened geometric formats. In this survey, we review the algorithms which extract simple geometric primitives from raw dense 3D data. After giving an introduction to these techniques, from the acquisition modality to the underlying theoretical concepts, we propose an application-oriented characterization, designed to help select an appropriate method based on one's application needs, and compare recent approaches. We conclude by giving hints for how to evaluate these methods and a set of research challenges to be explored. (more)

  A. Kaiser, J.A. Ybanez Zepeda and T. Boubekeur:
A Survey of Simple Geometric Primitives Detection Methods for Captured 3D Data
Computer Graphics Forum, 38(1):167-196, February 2019 (first published July 2018).     
(h5-index = 49 at time of publication)

  EUROGRAPHICS 2019, May 6-10, Genova, Italy  

Robust 3D Object Detection and Tracking on Mobile Devices

Master's Thesis. Work done while at Wikitude GmbH in collaboration with TU Wien.
Collaborators: Nicolas Goell, Markus Eder

Keywords: Augmented reality, Mobile devices, Scene tracking, Simultaneous Localization And Mapping (SLAM), RGB images, Point features, Edge features, Keyframes, SIMD on embedded CPU (ARM NEON).

Creation of a visual detection and tracking system based on Simultaneous Localization and Mapping (SLAM) for three dimensional objects on mobile devices, for augmented reality applications. Study of the state of the art and implementation of the detector in C++, inspired by existing methods.

DTIAtlasBuilder

Work done as an intern at the NIRAL within the University of North Carolina at Chapel Hill.
Collaborators: Dr. Martin Styner, Francois Budin

Keywords: Brain imaging, Diffusion Tensor Images (DTI), Magnetic Resonance Imaging (MRI), Tractography, 3D voxel registration, Qt, ITK, Graphical user interface, Cross-platform, 3DSlicer.

Development of a user interface for the creation of an atlas image of Diffusion Tensor Images. Cross-platform implementation of the interface in C++ and Python with Qt and ITK to execute the image processing pipeline. Integration to 3DSlicer as an extension.
Several contributions to open-source projects.

  A. R. Verde, F. Budin, J.B. Berger, A. Gupta, M. Farzinfar, A. Kaiser, M. Ahn, H. Johnson, J. Matsui, H. C. Hazlett, A. Sharma, C. Goodlett, Y. Shi, S. Gouttard, C. Vachet, J. Piven, H. Zhu, G. Gerig and M. Styner: UNC-Utah NA-MIC framework for DTI fiber tract analysis
Frontiers in Neuroinformatics, January 2014  
(h5-index = 26 at time of publication)

  2013 NA-MIC All Hands Meeting, Jan 7-11, Salt Lake City, UT, USA