Surface texture processing
Abstract/Contents
- Abstract
- Reconstruction and understanding the 3D environment is a fundamental step towards many applications including virtual reality, 3D vision, and robotics. A 3D environment is digitally represented by 3D geometry and color, where color information is usually stored as a surface texture. Our work focuses on processing the geometry and color information, so as to reconstruct and understand the surface texture. Specifically, our goal is to reconstruct high-quality surface textures on 3D surfaces in real environments captured by commodity 3D scanners, and to provide effective solutions for learning geometric and semantic information about the environment from surface textures. The main challenge in high-quality texture reconstruction is to deal with inconsistencies caused by scanning errors from geometry, camera poses, and by image artifacts coming from auto-exposure and motion blur. For understanding surface texture, we identify as core problems the definition of a canonical and regular parameterization space, the construction of this space with a robust and efficient solution, and the design of powerful convolution kernels in this space for learning effective features from the surface texture. This thesis explores techniques to address the texture acquisition problem from two different perspectives. In our first perspective, we propose to compensate for all the artifacts with flexible parametric models, for example, by explicitly warping the video frames and balancing the colors to achieve better consistency. Observing that motion blur is a ubiquitous artifact in many input video frames, we advocate replacing the naive averaging fusion methods with a best-view selection approach, based on pixel sharpness while enforcing boundary coherence by connecting regions from multiple frames. We show that explicit color consistency optimization and the novel fusion of input images can produce high-quality and sharp surface textures. However, it is hard to fully compensate for different sources of variance in the real environment (e.g., misaligned scanning geometry and complex lighting) with an explicit parametric model. Therefore in our second perspective, we propose to learn a deep metric that tolerates these errors instead of attempting to remove them, and uses the metric to guide a realistic texture generation without penalizing slight differences between the texture and the image observations. We model this deep metric as a patch-based discriminator using an image convolutional neural network (CNN) and jointly optimize it with the texture to obtain realistic surface appearance. While our deep metric can be implemented with an image CNN as a widely-studied deep learning technique, one extension in this thesis is to directly apply a CNN in the texture domain of a 3D surface. Although most works apply axis-aligned convolution kernels in images, there is no consensus on what is the best operator to apply on surface texture signals in 3D, mainly because of ambiguities in surface parameterization. Therefore, this thesis solves the key challenge by obtaining a consistent and canonical surface parameterization that maps the 3D surface into a 2D space where 2D convolutions can be applied. We find that this challenge is related to the seamless surface parameterization problem in the computational geometry community. While this problem has been studied for more than a decade, the existing state-of-the-art method is formulated as a mixed-integer programming problem (NP-hard), where an effective and robust solution is unavailable. We reformulate the problem by approximating it with a minimum-cost flow graph where there is a guaranteed global optimal solution within polynomial time complexity. With such a formulation, we obtain a robust and efficient, seamless parameterization for the geometry of a complex 3D environment. With a canonical surface parameterization, we propose a surface convolution operator that extracts effective features for 3D learning tasks. We show that such a surface CNN is powerful and descriptive for 3D semantic segmentation with canonical feature extraction. Specifically, we define a geodesic neighborhood and determine the local coordinate system based on this parameterization. Then, we propose a novel four-way rotationally symmetric convolution operator that canonicalizes feature extraction, bypassing the existence of orientation singularities in the parameterization. As a result, our surface convolution can efficiently handle high-resolution texture signals, and thereby outperform other less efficient dense 3D convolution operators. The canonical surface parameterization serves not only as a basis for surface convolution operators but also as an important feature that is highly-correlated with the color signals. From the existing scanning data with aligned color images and 3D geometry, we create a dataset by pairing the RGB images with pre-computed 3D canonical frames, given the camera transformation. We train a neural network to estimate the canonical frames from RGB images. With understanding the 3D canonical frames from RGB images, our network enables various applications including surface normal estimation, feature matching and augmented reality. In general, our proposed techniques for surface texture processing advance the state of the art in the visual quality of texture reconstruction and the accuracy for understanding from surface color signals. In the future, there is an opportunity to further study texture reconstruction and understanding as a joint problem, so that we can benefit from synergies between the two
Description
| Type of resource | text |
|---|---|
| Form | electronic resource; remote; computer; online resource |
| Extent | 1 online resource |
| Place | California |
| Place | [Stanford, California] |
| Publisher | [Stanford University] |
| Copyright date | 2020; ©2020 |
| Publication date | 2020; 2020 |
| Issuance | monographic |
| Language | English |
Creators/Contributors
| Author | Huang, Jingwei |
|---|---|
| Degree supervisor | Guibas, Leonidas J |
| Thesis advisor | Guibas, Leonidas J |
| Thesis advisor | Bohg, Jeannette, 1981- |
| Thesis advisor | Liu, Cheng-Yun Karen, 1977- |
| Degree committee member | Bohg, Jeannette, 1981- |
| Degree committee member | Liu, Cheng-Yun Karen, 1977- |
| Associated with | Stanford University, Computer Science Department. |
Subjects
| Genre | Theses |
|---|---|
| Genre | Text |
Bibliographic information
| Statement of responsibility | Jingwei Huang |
|---|---|
| Note | Submitted to the Computer Science Department |
| Thesis | Thesis Ph.D. Stanford University 2020 |
| Location | electronic resource |
Access conditions
- Copyright
- © 2020 by Jingwei Huang
- License
- This work is licensed under a Creative Commons Attribution 3.0 Unported license (CC BY).
Also listed in
Loading usage metrics...