NAS-NeRF: Generative Neural Architecture Search for Neural Radiance Fields

Saeejith Nair1, Yuhao Chen1, Mohammad Javad Shafiee1,2,3, Alexander Wong1,2,3
1University of Waterloo,  2Darwin AI,  3Waterloo Artificial Intelligence Institute
Paper arXiv Code

Abstract

Neural radiance fields (NeRFs) enable high-quality novel view synthesis, but their high computational complexity limits deployability. While existing neural-based solutions strive for efficiency, they use one-size-fits-all architectures regardless of scene complexity. The same architecture may be unnecessarily large for simple scenes but insufficient for complex ones. Thus, there is a need to dynamically optimize the neural network component of NeRFs to achieve a balance between computational complexity and specific targets for synthesis quality. We introduce NAS-NeRF, a generative neural architecture search strategy that generates compact, scene-specialized NeRF architectures by balancing architecture complexity and target synthesis quality metrics. Our method incorporates constraints on target metrics and budgets to guide the search towards architectures tailored for each scene. Experiments on the Blender synthetic dataset show the proposed NAS-NeRF can generate architectures up to 5.74× smaller, with 4.19× fewer FLOPs, and 1.93× faster on a GPU than baseline NeRFs, without suffering a drop in SSIM. Furthermore, we illustrate that NAS-NeRF can also achieve architectures up to 23× smaller, with 22× fewer FLOPs, and 4.7× faster than baseline NeRFs with only a 5.3% average SSIM drop.



Hover over image to move the zoomed in patch; Click on ground-truth image to switch to a different image (best viewed on desktop).

Ground Truth
NeRF
NAS-NeRF S
NAS-NeRF XS
NAS-NeRF XXS


NeRF
NAS-NeRF S
NAS-NeRF XS
NAS-NeRF XXS
NeRF
NAS-NeRF S
NAS-NeRF XS
NAS-NeRF XXS


NAS-NeRF Field Cell Architecture

The NAS-NeRF pipeline is composed of all the components of the classic NeRF pipeline, along with two NAS-NeRF field cells in series for performing the coarse and fine hierarchical sampling. The nature of our parameterization ensures that the NAS-NeRF field cell can be plugged into most other NeRF methods, as our optimizations revolve entirely around the core network architectures.

NAS-NeRF Field Cell




Generated Architectures

Architecture efficiency ratio vs. synthesis quality (SSIM) (top) and inference speed (bottom), marker size ∝ parameter count. Here, architecture efficiency ratio is a measure of the number of FLOPs required for inference on the baseline NeRF architecture relative to the generated NAS-NeRF architecture.

Architecture Efficiency Ratio vs. Synthesis Quality


BibTeX

@article{nair2023nerf,
  author    = {Nair, Saeejith and Chen, Yuhao and Shafiee, Mohammad Javad and Wong, Alexander},
  title     = {NAS-NeRF: Generative Neural Architecture Search for Neural Radiance Fields},
  journal   = {arXiv preprint arXiv:2309.14293},
  year      = {2023},
}