Main content
Top content
Structure from Neural Networks (SfN²)
Abstract
For 3D reconstruction from a set of images or video frames, the Structure from Motion (SfM) approach is still the dominant method. The SfM reconstruction pipeline has become quite robust and accurate. Recently, artificial neural networks (ANNs) have been successfully applied for 3D reconstruction of a priori known object categories. We want to go beyond the restriction of pre-defined object categories by emulating SfM using of ANNs for reconstruction of arbitrary 3D objects. We have developed a synthetic scalable cube dataset for training, testing and validating ANNs. We will show that it is possible to reconstruct 3D shapes by ANNs, and we will benchmark three different ANN architectures inspired by the three most important strategies of SfM pipelines --- the global, hierarchical, and incremental key point matching. Along with this benchmark, we introduce our dataset and its generator and discuss how it can be used for capturing the complexity of real world images.
Dataset
| Cube Setup | Link | ||||||
 |  |  |  | ... |  | 3x3x3 100 000 cubes | Download - jpg views + txt Download - obj files |
 |  |  |  | ... |  | 4x4x4 300 000 cubes | Download - jpg views + txt Download - obj files |
 |  |  |  | ... |  | 8x8x8 430 000 cubes | Download - jpg views + txt Download - obj files |
| This work is licensed under a Creative Commons Attribution 4.0 International License. For giving appropriate credit, cite one of the publication from the references below. |  | ||||||
Generator Tools
Python Voxelizer
This pyhton script create voxelized objects incl. a voxel set list out of ply, off or stl 3D object files.
Cube Generator
This generator, written in Matlab, randomly generates n 3D objects. Each such object is created by taking a unit cube in R^3 and subdividing it into a r x r x r subcubes. The parameter r can be defined by the user. By ensuring the uniqueness of the cube distribution in the voxel grid, this generator is able to generate 2^(r^3) different 3D objects and export them as 3D *.obj object files.
Views Generator
This generator, written in Matlab, (can be optionally used for voxelization of 3D objects and) renders w input images with a pixel resolution x by x. Where the w different viewpoints are uniformly distributed around the object by using the Fibonacci lattice
Visualized Results
Visualization of the accuracy of the global, hierarchical, and incremental over all 1000 epochs on the training samples. Filter size of each convolutional layer = 32.
Visualization of the accuracy of the global, hierarchical, and incremental over all 1000 epochs on the validation samples. Filter size of each convolutional layer = 32.
Visualization of the accuracy of the global, hierarchical, and incremental over all 1000 epochs on the training samples. Filter size of each convolutional layer = 16.
Supplementary Plots and Network Architectures
Download supplementary Plots of the network performance and network architectures. PDF
