[summary] Occupancy Networks: Learning 3D Reconstruction in Function Space (2019)

[arxiv, papers with code, supplement]

Summary

Approach

Instead of reconstructing a 3D shape in the form of a (discrete) voxel grid, point cloud, or mesh from the input data, occupancy networks return a function that predicts an occupancy probability for a continuous 3D point in R³:

The trick is the following: a function that takes an observation x as input and returns a function mapping a 3D point p to the occupancy probability can be replaced by a function that takes a pair (p, x) and returns the occupancy probability (see also: uncurrying):

Limitations of existing 3D data representations

• Voxel grids: memory-heavy, up to 128³–256³ maximum resolution
• Point clouds: need post-processing to generate a (mesh) surface
• Meshes: existing approaches often require additional regularization, can generate only meshes with simple topology, need the same class reference template, or cannot guarantee closed surfaces

Pipeline

• Training: the network f_θ(p, x) takes as input the task specific object (for example, for single image 3D reconstruction this would be an RGB image) and a batch of 3D points, randomly sampled from the ground truth 3D representation. For each 3D point, the network predicts the occupancy probability which is then compared with the ground truth to compute the mini-batch loss:
• Inference: to extract a 3D mesh from the learned f_θ(p, x), the paper uses a Multiresolution IsoSurface Extraction (MISE) algorithm, which as a first step is building an octree by progressively sampling in the points where neighbors occupancy predictions do not match. After that, the Marching Cubes algorithm is applied to extract the mesh surface. Finally, the mesh is refined even further using the Fast-Quadric-Mesh-Simplification algorithm.

Network architecture

• The network architecture is generally the same across different tasks (e.g. single image 3D reconstruction or point cloud completion) with the task-specific encoder being the only changing element.
  
• Once the task encoder has produced the embedding c, it is passed as input along with the batch of T sampled 3D points, which are processed by 5 sequential ResNet blocks. To condition the network output on the input embedding c, Conditional Batch Normalization is used:
  
• For single image 3D reconstruction the network uses a ResNet-18 encoder with altered last layer to produce 256-dim embeddings.
  
• For point cloud completion a modified version of the PointNet encoder is used.
  
• For voxel super-resolution the network uses a 3D CNN encoder, which encodes a 32³ input into a 256-dim embedding vector.
  

Experiments

• Single image 3D reconstruction:
  
• Point cloud completion:
  
• Voxel super resolution:
  
• Generalization to the Pix3D dataset (unseen data)—could be better:
  

Metrics

• Comparison against 3D-R2N2, PSGN, Pix2Mesh and AtlasNet on the ShapeNet dataset.
  

Used diagrams are taken from the paper with minor changes.
All errors present in the interpretation are my own.