[ Neural rendering] Neural Geometry And Rendering Challenge@eccv2022

Neural Geometry and Rendering (NGR): Advances and the Common Objects in 3D (CO3D) Challenge @ECCV2022

We took part in the NGR-CO3D challenge hosted by ECCV2022. As a result, we placed the winning entry for the “manyview”-reconstruction track.

Task Definition

Given a sequence of RGB images, ROI-masks, and camera-poses, one’s algorithm have to output appearance (RGB), depth-map (D), and ROI-mask (A) for given test camera poses for distinct 88 video sequences in total. Consequently, total number of 3D scene representation networks (NeRF in our case), should be trained, is 88.

Our Apporach

We employed a simple ensemble representation of an object with the NeRF and TensoRF. To prevent overfitting phenomena, we applied per-sample-entropy, L1-sparsity, and total-variation loss to our objectives. To train all of 88 models at once, we used the Multi-node Distributed Data Parallel Package shipped in the PyTorch.

Quantitative Results

Until the evaluation server shut down, we slightly but granularly imporved the performance of our method.

Qualitative Results

Here are showcases of our novel-view synthesis (order in predicted RGB, Depth, and ROI).

Keywords:

Neural Rendering, Neural Radiance Field (NeRF), Tensorial Radiance Field (TensoRF), Implicit Volume Rendering, Novel-View Synthesis, 3D Reconstruction

Check out our presentation slide: InvitedTalk_winning_entry_manyview, NGR-CO3D@ECCV, 2022