SS3DM: Benchmarking Street-View Surface Reconstruction with a Synthetic 3D Mesh Dataset
NeurIPS 2024 Dataset and Benchmark Track
[Paper] [SS3DM-Exporter] [SS3DM-Benchmark]Abstract
Reconstructing accurate 3D surfaces for street-view scenarios is crucial for applications such as digital entertainment and autonomous driving simulation. However, existing street-view datasets, including KITTI, Waymo, and nuScenes, only offer noisy LiDAR points as ground-truth data for geometric evaluation of reconstructed surfaces. These geometric ground-truths often lack the necessary precision to evaluate surface positions and do not provide data for assessing surface normals. To overcome these challenges, we introduce the SS3DM dataset, comprising precise Synthetic Street-view 3D Mesh models exported from the CARLA simulator. These mesh models facilitate accurate position evaluation and include normal vectors for evaluating surface normal. To simulate the input data in realistic driving scenarios for 3D reconstruction, we virtually drive a vehicle equipped with six RGB cameras and five LiDAR sensors in diverse outdoor scenes. Leveraging this dataset, we establish a benchmark for state-of-the-art surface reconstruction methods, providing a comprehensive evaluation of the associated challenges.
Benchmarking Results
| Method | IoU↑ | Prec.↑ | Recall↑ | F-score↑ | Acc↓ | Comp↓ | CD↓ | Acc_N↓ | Comp_N↓ | CD_N↓ | CD+CD_N↓ |
|---|---|---|---|---|---|---|---|---|---|---|---|
| R3D3 | 0.003 | 0.006 | 0.008 | 0.007 | 0.898 | 0.925 | 1.823 | 0.717 | 0.712 | 1.429 | 3.252 |
| UrbanNeRF | 0.046 | 0.086 | 0.123 | 0.098 | 0.432 | 0.575 | 1.007 | 0.442 | 0.557 | 0.999 | 2.006 |
| SuGaR | 0.032 | 0.069 | 0.053 | 0.056 | 0.444 | 0.469 | 0.914 | 0.650 | 0.662 | 1.312 | 2.226 |
| StreetSurf (RGB) | 0.044 | 0.078 | 0.067 | 0.069 | 0.372 | 0.490 | 0.862 | 0.517 | 0.616 | 1.133 | 1.995 |
| NeRF-LOAM | 0.072 | 0.107 | 0.139 | 0.116 | 0.151 | 0.400 | 0.551 | 0.687 | 0.724 | 1.411 | 1.962 |
| StreetSurf (LiDAR) | 0.107 | 0.206 | 0.245 | 0.215 | 0.246 | 0.367 | 0.613 | 0.506 | 0.582 | 1.088 | 1.701 |
| StreetSurf (Full) | 0.116 | 0.196 | 0.218 | 0.198 | 0.202 | 0.367 | 0.569 | 0.414 | 0.541 | 0.955 | 1.524 |