CODEWITHSUNDEEP
machine-learningdeep-learningcomputer-vision
Hongyu Shen
Hongyu Shen

Reputation: 31

What is NeRF(Neural Radiance Fields) used for?

Recently I am studying the research NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis(https://www.matthewtancik.com/nerf), and I am wondering: What is it used for? Will there be any application of NeRF?

The result of this technique is very impressive, but what is it used for? I keep thinking of this question over and over again. It is very realistic, the quality is perfect, but we don't want to see the camera swinging around all the time, right?

Personally, this technique has some limitations:

  1. Cannot generate views that never seen in input images. This technique interpolates between two views.

  2. Long training and rendering time: According to the authors, it takes 12 hours to train a scene, and 30s to render one frame.

  3. The view is static and not interactable.

I don't know if it is appropriate to compare NeRF with Panorama and 360° image/video, essentially they are different, only NeRF uses deep learning to generate new views, the others basically are just using smart phone/camera to capture scenes plus some computer vision techniques. Still, the long training time makes NeRF less competitive in this application area. Am I correct?

Another utility I can think of is product rendering, however, NeRF doesn't show advantages compare to using 3D software to render. Like commercial advertisement, usually it requires animation and special effects, then definitely 3D software can do better.

The potential use of NeRF might be 3D reconstruction, but that would be out of the scope, although it is able to do that. Why do we need to use NeRF for 3D reconstruction? Why not use other reconstruction techniques? The unique feature of NeRF is the ability of creating photo-realistic views, if we use NeRF for 3D reconstruction, then this feature becomes pointless.

Does anyone have new ideas? I would like to know.

Upvotes: 2

Views: 993

Answers (1)

Roman Shapovalov
Roman Shapovalov

Reputation: 2805

Why do we need to use NeRF for 3D reconstruction?

The alternative would be multi-view stereo, which produces point clouds of finite resolution and is susceptible to illumination changes. If you then render such point cloud without non-trivial post-processing, it will not look photorealistic.

I don't know if it is appropriate to compare NeRF with Panorama and 360° image/video,

Well, if you deal with exactly flat scene with simple lighting (i.e. ambient light and Lambertian objects), then you can use panorama techniques for new view synthesis. In general though, it won’t produce the result you expect. You have to know the depth to interpolate correctly.

When it comes to practical limitations (slow; does not model deformations), NeRF should be considered a milestone that provided a proof of concept that representing surface as a level set of MLP-modelled function can result in sharp rendering. There is already good progress in addressing those limitations, and multiple works apply this idea for practical tasks.

Upvotes: 1

Related Questions